## org.eclipse.datatools.sqltools.data.internal.core.load
Class DataFileTokenizer

    java.lang.Object
      org.eclipse.datatools.sqltools.data.internal.core.load.DataFileTokenizer

    public class DataFileTokenizer

extends java.lang.Object

Constructor Summary
`DataFileTokenizer(java.lang.String filePath, java.lang.String delims)`

Method Summary
`boolean` `bomEncoding()` The BOM (Byte Order Marks) is a Unicode special marker placed at the top of the file that indicate its encoding.
`void` `close()`
`void` `consume(java.lang.String s)`
`java.lang.String` `nextToken()`
`void` `open()` Opens the extracted file to be loaded into the table.
`java.lang.String` `peek()`

Methods inherited from class java.lang.Object
`equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Constructor Detail

### DataFileTokenizer

    public DataFileTokenizer(java.lang.String filePath,
                             java.lang.String delims)

Method Detail

### bomEncoding

    public boolean bomEncoding()
                        throws java.io.IOException

  The BOM (Byte Order Marks) is a Unicode special marker placed at the top of the file that indicate its encoding. The UTF-8 representation of the BOM is the byte sequence EF BB BF. The BOM is optional for UTF-8. Before loading the file, a check is made if BOM are introduced in the UTF-8 encoded file. If the first three bytes are 0xEF, 0xBB, 0xBF the first character is skipped as it is BOM.

  Throws:
    `java.io.IOException`

### open

    public void open()
              throws java.io.IOException

  Opens the extracted file to be loaded into the table. The method assumes the file was saved using UTF-8 character encoding

  Throws:
    `java.io.IOException`

### peek

    public java.lang.String peek()
                          throws java.io.IOException

  Throws:
    `java.io.IOException`

### consume

    public void consume(java.lang.String s)
                 throws java.lang.Exception

  Throws:
    `java.lang.Exception`

### nextToken

    public java.lang.String nextToken()
                               throws java.io.IOException

  Throws:
    `java.io.IOException`

### close

    public void close()
               throws java.io.IOException

  Throws:
    `java.io.IOException`