Java 识别令牌

Question

我正在执行一项任务，我必须查看文件并识别令牌。我认为我走在正确的轨道上。我有大部分工作，但我无法识别特定字符，因此我可以在识别令牌时选择忽略它。它是\n 我得出了这个结论，因为当我尝试在 if 语句中执行 tokens.get(6).equals('\n') 时，它出现的结果就像它不是 \n 一样错误，我无法识别存储在这个数组中的内容指数值。文本文件如下所示

编辑：我知道它是\n，因为我做了System.out.println("ddd" + tokens.get(6) +"dddd")，输出是ddd，然后是一个新行和dddd

编辑 2：我添加了 System.out.println(", unidenfified " + word);这是输出

read a
read b
c := a + b + 3
write c 

输出应该是这样的

<read>, read
<id>, a
<read>, read
<id>, b
<id>, c
<assign>, :=
<id>, a
<add_op>, +
<id>, b
<add_op>, +
<number>, 3
<write>, write
<id>, c 

我的输出是这样的

<read>, read
<error>, unidenfified
<error>, unidenfified
<assign>, :=
<id>, a
<add_op>, +
<id>, b
<add_op>, +
<error>, unidenfified
<error>, unidenfified

我不确定为什么会这样。 我的代码：

import java.util.ArrayList;
import java.util.Scanner;
import java.io.*;
import java.lang.Character;

public class Tokens {

    public static void main(String[] args) throws IOException {

        Scanner input = new Scanner(System.in); // Scanner for taking input from
                                                // the user

        String fileName;
        System.out.println("Enter the name of the file.");
        fileName = input.next();

        fileExists(fileName); // Checks to see if the file exists

        ArrayList<Character> arrayOfTokens = new ArrayList<Character>();
        ArrayList<String> assembled = new ArrayList<String>();
        readToArray(arrayOfTokens, fileName);
        assembled = assembleTokens(arrayOfTokens);

        for(int i = 0; i < assembled.size(); i++) {
             analyze(assembled.get(i));
         }

    }

    /*
     * readToArray goes through a file and adds all its elements in individual
     * character form. It is stored into an arraylist and it is then returned
     * 
     * @param storeChar: This is an arraylist of characters that the characters
     * will be saved into and then returned.
     * 
     * @param fileName: The filename that you want to take the data from.
     */
    private static ArrayList<Character> readToArray(
            ArrayList<Character> storeChar, String fileName) throws IOException {
        /*
         * Block of code to setup the fileInput stream to take in data from the
         * file. Reads character by character and stores into an arraylist. int
         * atChar: the current character the reader is at. Returns in int format
         * (Need to be converted to character later on) int currentIndex: to add
         * a character to an index. Increments until no more characters are left
         */
        FileInputStream fileInput = new FileInputStream(fileName);
        int atChar;
        int currentIndex = 0;

        /*
         * Loop to go through and add the converted character from an int to the
         * arraylist. Loops until atChar returns -1 which means no more
         * characters in file.
         */
        while ((atChar = fileInput.read()) != -1) {
            storeChar.add(currentIndex, (char) (atChar));
            currentIndex++;
        }
        fileInput.close();

        return storeChar;
    }

    /*
     * fileExists method makes sure the file the user enters exists in the
     * system. If it doesn't then the program will terminate before any further
     * code is executed.
     * 
     * @param fileName: Takes in a string paramater of the file name that you
     * want to if it exists.
     */
    private static void fileExists(String fileName) {

        boolean ifExists; // Boolean statement that will later be set to the
                            // value of whether the file exists or not

        File file = new File(fileName);
        ifExists = file.exists();

        if (ifExists == false) {
            System.out
                    .println("Unable to find the file. Will now close the program.");
            System.exit(0);
        }
    }

    private static ArrayList<String> assembleTokens(ArrayList<Character> tokens) {

        ArrayList<String> identified = new ArrayList<String>();
        int counter = 0;
        String concatinated = "";

        while (counter < tokens.size()) {
            if (!tokens.get(counter).equals(' ')) {
                concatinated += tokens.get(counter);
                counter++;
            } else {
                identified.add(concatinated);
                concatinated = "";
                counter++;
            }
        }

        return identified;
    }

    private static void analyze(String word) {
        if(word.equals("read")) {
            System.out.println("<read>, read");
        } else if(word.equals("write")) {
            System.out.println("<write>, write");
        } else if(word.equals(":=")) {
            System.out.println("<assign>, :=");
        } else if(word.equals("(")) {
            System.out.println("<lparen>, (");
        } else if(word.equals(")")) {
            System.out.println("<rparen>, )");
        } else if(word.equals("+") || word.equals("-")) {
            System.out.println("<add_op>, " + word);
        } else if(word.equals("*") || word.equals("/") || word.equals("//") || word.equals("%")) {
            System.out.println("<mult_op>, " + word);
        } else if(word.matches("[a-z]+[A-Za-z0-9]*")) {
            System.out.println("<id>, " + word);
        } else if(word.matches("\\d+(\\.\\d+)?")) {
            System.out.println("<number>, " + word);
        } else {
            System.out.println("<error>, unidenfified");
        }

    }
}

Answer 1

读取文件时只需过滤掉\n

 if (atChar != '\n') {
     storeChar.add(currentIndex, (char) (atChar));
 }

你也可以使用

fileInput.readLine()

然后标记字符串

编辑

看看你上次的编辑，如果最好做一个readLine，然后使用String.split(" ");来获取你的令牌