com.jidesoft.editor.tokenmarker
Class TokenMarker

java.lang.Object
  com.jidesoft.editor.tokenmarker.TokenMarker

Direct Known Subclasses:

BatchFileTokenMarker, CTokenMarker, EiffelTokenMarker, HTMLTokenMarker, MakefileTokenMarker, PatchTokenMarker, PerlTokenMarker, PHPTokenMarker, PlainTokenMarker, PropsTokenMarker, PythonTokenMarker, ShellScriptTokenMarker, SQLTokenMarker, TeXTokenMarker, VerilogTokenMarker, VHDLTokenMarker, XMLTokenMarker

public abstract class TokenMarker
extends Object

A token marker splits lines of text into tokens. Each token carries a length field and an identification tag that can be mapped to a color for painting that token.

For performance reasons, the linked list of tokens is reused after each line is tokenized. Therefore, the return value of markTokens should only be used for immediate painting. Notably, it cannot be cached.

Version:

$Id: TokenMarker.java,v 1.32 1999/12/13 03:40:30 sp Exp $

Author:

Slava Pestov

See Also:

Token

Nested Class Summary

class

TokenMarker.LineInfo Deprecated. replaced by a byte array and an Object array in TokenMarker to reduce memory usage.

Field Summary

protected Object[]	_lineInfoExtended An array for storing other information about lines.
protected byte[]	_lineInfoToken An array for storing token information about lines.
protected Token[]	_tokens Token array to contain firstToken and lastToken so that the token marker process is thread safe.
protected Token	firstToken Deprecated. replaced by the first element of _tokens
protected int	lastLine The last tokenized line.
protected Token	lastToken Deprecated. replaced by the second element of _tokens
protected int	length The number of lines in the model being tokenized.
protected TokenMarker.LineInfo[]	lineInfo Deprecated. replaced by _lineInfoToken && _lineInfoExtended to reduce memory usage.
protected boolean	nextLineRequested True if the next line should be painted.

Constructor Summary

protected

TokenMarker() Creates a new TokenMarker.

Method Summary

protected void	addToken(int length, byte id) Adds a token to the token list.
void	deleteLines(int index, int lines) Informs the token marker that line have been deleted from the document.
protected void	ensureCapacity(int index) Ensures that the _lineInfoToken array can contain the specified index.
SyntaxDocument	getDocument() Get the document the TokenMarker is parsing.
byte	getLastToken(int lineIndex) Get the last token of the line.
int	getLineCount() Returns the number of lines in this token marker.
void	insertLines(int index, int lines) Informs the token marker that lines have been inserted into the document.
boolean	isNextLineRequested() Returns true if the next line should be repainted.
Token	markTokens(Segment line, int lineIndex) A wrapper for the lower-level markTokensImpl method that is called to split a line up into tokens.
protected abstract byte	markTokensImpl(byte token, Segment line, int lineIndex) An abstract method that splits a line up into tokens.
void	setDocument(SyntaxDocument document) Set the document the TokenMarker is parsing.
boolean	supportsMultilineTokens() Returns if the token marker supports tokens that span multiple lines.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

firstToken

@Deprecated
protected Token firstToken

Deprecated. replaced by the first element of _tokens

The first token in the list. This should be used as the return value from markTokens().

lastToken

@Deprecated
protected Token lastToken

Deprecated. replaced by the second element of _tokens

The last token in the list. New tokens are added here. This should be set to null before a new line is to be tokenized.

_tokens

protected final Token[] _tokens

Token array to contain firstToken and lastToken so that the token marker process is thread safe. The first element is firstToken while the second is lastToken.

lineInfo

@Deprecated
protected TokenMarker.LineInfo[] lineInfo

Deprecated. replaced by _lineInfoToken && _lineInfoExtended to reduce memory usage.

An array for storing information about lines. It is enlarged and shrunk automatically by the insertLines() and deleteLines() methods.

_lineInfoToken

protected byte[] _lineInfoToken

An array for storing token information about lines. It is enlarged and shrunk automatically by the insertLines() and deleteLines() methods.

_lineInfoExtended

protected Object[] _lineInfoExtended

An array for storing other information about lines. It's not initialized in TokenMarker. If you want to use this field, please initialize/enlarge/shrink it at the insertLines() and deleteLines() methods.

length

protected int length

The number of lines in the model being tokenized. This can be less than the length of the _lineInfoToken array.

lastLine

protected int lastLine

The last tokenized line.

nextLineRequested

protected boolean nextLineRequested

True if the next line should be painted.

Constructor Detail

TokenMarker

protected TokenMarker()

Creates a new TokenMarker. This DOES NOT create a _lineInfoToken array; an initial call to insertLines() does that.

Method Detail

markTokens

public Token markTokens(Segment line,
                        int lineIndex)

A wrapper for the lower-level markTokensImpl method that is called to split a line up into tokens.

Parameters:

line - The line

lineIndex - The line number

Returns:

the tokens.

getLastToken

public byte getLastToken(int lineIndex)

Get the last token of the line.

Parameters:

lineIndex - the line index

Returns:

the last token of the line.

markTokensImpl

protected abstract byte markTokensImpl(byte token,
                                       Segment line,
                                       int lineIndex)

An abstract method that splits a line up into tokens. It should parse the line, and call addToken() to add syntax tokens to the token list. Then, it should return the initial token type for the next line.

For example if the current line contains the start of a multiline comment that doesn't end on that line, this method should return the comment token type so that it continues on the next line.

Parameters:

token - The initial token type for this line

line - The line to be tokenized

lineIndex - The index of the line in the document, starting at 0

Returns:

The initial token type for the next line

supportsMultilineTokens

public boolean supportsMultilineTokens()

Returns if the token marker supports tokens that span multiple lines. If this is true, the object using this token marker is required to pass all lines in the document to the markTokens() method (in turn).

The default implementation returns true; it should be overridden to return false on simpler token markers for increased speed.

Returns:

true by default.

insertLines

public void insertLines(int index,
                        int lines)

Informs the token marker that lines have been inserted into the document. This inserts a gap in the _lineInfoToken array.

Parameters:

index - The first line number

lines - The number of lines

deleteLines

public void deleteLines(int index,
                        int lines)

Informs the token marker that line have been deleted from the document. This removes the lines in question from the _lineInfoToken array.

Parameters:

index - The first line number

lines - The number of lines

getLineCount

public int getLineCount()

Returns the number of lines in this token marker.

Returns:

the line count.

isNextLineRequested

public boolean isNextLineRequested()

Returns true if the next line should be repainted. This will return true after a line has been tokenized that starts a multiline token that continues onto the next line.

Returns:

true if the next line should be repainted. Otherwise false.

ensureCapacity

protected void ensureCapacity(int index)

Ensures that the _lineInfoToken array can contain the specified index. This enlarges it if necessary. No action is taken if the array is large enough already.

It should be unnecessary to call this under normal circumstances; insertLine() should take care of enlarging the line info array automatically.

Parameters:

index - The array index

addToken

protected void addToken(int length,
                        byte id)

Adds a token to the token list.

Parameters:

length - The length of the token

id - The id of the token

getDocument

public SyntaxDocument getDocument()

Get the document the TokenMarker is parsing.

Returns:

the document.

setDocument

public void setDocument(SyntaxDocument document)

Set the document the TokenMarker is parsing.

This method is invoked automatically by SyntaxDocument.setTokenMarker(TokenMarker). You should always invoke that method to build correct relationship between SyntaxDocument and TokenMarker.

Parameters:

document - the document