Delimiter
In computing, a delimiter is a character or a sequence of characters for specifying the boundary between separate, independent regions in data such as a text file or data stream.[1][2]
In mathematics, delimiters are often used to specify the scope of an operation in an expression, and can occur both as isolated symbols (e.g., colon in "") and as a pair of opposing-looking symbols (e.g., angled brackets in ).
Delimiters aren't the only way to delineate data boundaries. For example, declarative notation indicates the length of a field at the start of the field instead of relying on delimiters.[3]
Examples
[edit]Delimiters are used for a wide range of purposes. The following examples demonstrate a small fraction of their applicability.
Tabular data
[edit]Tabular data, organized as rows and columns, is often delimited. A field delimiter separates the columns of a row, with each column corresponding to a field in that row, and a record delimiter separates the rows, with each row corresponding to a record.[4] The commonly used comma-separated values (CSV) format uses a comma to delimit fields, and an newline to delimit records. The following CSV data represents three records each with four fields. The first line is metadata that names the fields.
fname,lname,age,salary nancy,davolio,33,$30000 erin,borakova,28,$25250 tony,raphael,35,$28700
The related alternative tab-separated values (TSV) format uses the tab character instead of the comma to delimit fields. CSV and TSV formats are specific examples of a delimiter-separated values (DSV) format. A file saved in a DSV format can be classified as a flat-file database.
Bracket delimiters
[edit]Bracket delimiters, also called block delimiters, region delimiters, or balanced delimiters, mark the start and end of a region of text.[5][6] Commonly used bracket delimiters include:[7]
| Delimiters | Description |
|---|---|
( ) |
Parentheses; Lisp code is cited as recognizable by its use of parentheses[8] |
{ } |
Braces; also called curly brackets[9] |
[ ] |
Square brackets; commonly used to denote a subscript |
< > |
Angle brackets[10] |
" " |
Double quote; commonly used to denote a string literal[11] |
' ' |
Single quote; commonly used to denote a string literal or character literal[11] |
<? ?> |
Used in XML to denote a processing instruction[12] |
/* */ |
Used in many programming languages to denote a comment[13] |
<!-- --> |
Used in SGML, HTML, and XML to denote a comment |
<% %> |
Used in some web templates to specify a language boundary[14] |
ASCII information separator characters
[edit]The ASCII character set designates the following 4 non-textual control characters in the C0 code range from 28 to 31 as "Information Separators", which may be used in a hierarchical or in a general (non-hierarchical) manner:[15][16]
| ISO 646:1983 general name,
intended for application-specific use (may be non-hierarchical) |
ASCII 1986 name,
intended for hierarchical use[note 1] |
ASCII & UTF-8 code[note 2] | Unicode "Control Picture"[note 3] | |||||
|---|---|---|---|---|---|---|---|---|
| dec. | hex. | code | glyph | |||||
| INFORMATION SEPARATOR FOUR | IS4 | FILE SEPARATOR | FS | End of file or between a concatenation of files | 28 | 0x1C | U+241C | ␜ |
| INFORMATION SEPARATOR THREE | IS3 | GROUP SEPARATOR | GS | Between sections of data; not needed in simple data files | 29 | 0x1D | U+241D | ␝ |
| INFORMATION SEPARATOR TWO | IS2 | RECORD SEPARATOR | RS | End of a record or row | 30 | 0x1E | U+241E | ␞ |
| INFORMATION SEPARATOR ONE | IS1 | UNIT SEPARATOR | US | Between fields of a record, or members of a row | 31 | 0x1F | U+241F | ␟ |
Formats using ASCII information separators
[edit]1969 Internet Engineering Task Force (IETF) Request for Comments (RFC) #20 "ASCII format for Network Interchange" suggests 7-bit ASCII and uses these 4 Information Separators hierarchically.[18]
1986 National Institute of Standards and Technology (NIST) "Fingerprint Identification - Data Format for Information Interchange" also hierarchically delineates with these 4 separators.[19]
2007 ISO barcode standard GS1-128 uses either the special FNC1 (Function Code 1) flag or the ASCII GROUP SEPARATOR character after a variable length element.[20][21]
2015 RFC #7464 "JavaScript Object Notation (JSON) Text Sequences" encodes multiple JSON texts in UTF-8, each prefixed by the ASCII RECORD SEPARATOR and ending with an ASCII LINE FEED character.[22]
2024 IETF draft format "Unicode Separated Values (USV)"[23] allows substituting ASCII separator codes with their corresponding Unicode Control Picture (␜, ␝, ␞, ␟) to facilitate freeform Markdown-like text editing "in any typical modern editor, font, terminal, shell, search" using visible copy-pastable characters, an advantage over using ASCII's separators which aren't easily inputted and viewable in plain text editors.[17][24] This substitution makes USV both machine- and human-readable and editable.
Delimiter collision
[edit]Delimiter collision describes a limitation of using delimiters. When content information contains a delimiter, then the processing of the data will fail since the embedded delimiter will be incorrectly interpreted as a data boundary unless provisions are made to prevent the collision.[4][25] In XML, for example, collision can occur when content contains an angle bracket (< or >).
Each delimiter in a format can result in collision. In CSV, for example, field collision can occur when a field contains a comma (e.g., salary = "$30,000"), and record delimiter collision can occur when a field contains a newline. Both record and field delimiter collision occur frequently in CSV data.
A malicious user may seek to exploit collision. Consequently, delimiter collision can be the source of security vulnerability and exploit. Well-known examples include SQL injection and cross-site scripting in the context of SQL and HTML, respectively.
Solutions
[edit]Multiple methods for avoiding collision have been devised.
Prohibit use of specified delimiters in content
[edit]One solution to avoid the risk of delimiter collision is to prohibit content from containing a specified set of designated delimiters. A downside is arbitrary binary data can't be communicated without some workaround to handle data that contains those delimiter codes.
Obfuscation
[edit]Using a delimiter that is unlikely to appear in the content is an ad hoc approach that leads to limited success. It requires knowledge of expected content, guessing what won't appear in the content, and offers little security against malicious collisions.
Escape sequence
[edit]A commonly used method for avoiding delimiter collision is to use an escape character or a multi-character escape sequence. A designated character (e.g. the backslash \), or sequence of characters before a character that otherwise would indicate a boundary indicates that the delimiter character is not to be treated as a boundary.
Although effective, this technique has drawbacks including:
- Content can be hard to read when it contains numerous escape sequences, a problem referred to as leaning toothpick syndrome (due to use of \ to escape / in Perl regular expressions, leading to sequences such as "\/\/");
- Data becomes difficult to parse via regular expression
- Requires a way to escape the escape sequence (a way to use the escape sequence as content)
- An escape sequence can be cryptic to those unfamiliar with the syntax[26]
- The method does not protect against injection attacks[citation needed]
Higher level encoding
[edit]Some systems allow any character to be represented as a sequence of characters. This allows text that otherwise is a delimiter to be encoded in the content indirectly and thus prevent delimiter collision. A drawback of this method is that character codes are relatively hard to read, understand and memorize.
For example, Perl allows a character to be encoded as the sequence \x## where ## is the numeric value of the character code. The following shows how the sequence for double-quote (\x22) can be used to prevent collision with the delimiter that marks the begin and end of a string literal.
print "Nancy said \x22Hello World!\x22 to the crowd.";
produces the same output as:
print "Nancy said \"Hello World!\" to the crowd."; ### use escape char
Dual quoting delimiters
[edit]In contrast to escape sequences and escape characters, dual delimiters provide yet another way to avoid delimiter collision. Some languages, for example, allow the use of either a single quote ' or a double quote " to specify a string literal. For example, in Perl:
print 'Nancy said "Hello World!" to the crowd.';
produces the desired output without requiring escapes. This approach, however, only works when the string does not contain both types of quotation marks.
Doubling the delimiter
[edit]Visual Basic uses a single double quote " as a delimiter for the start and end of a string. To avoid collision when having a double quote " inside of a string, two consecutive double-quotes "" are used, so:[27]
print "Nancy said ""Hello World!"" to the crowd."
produces the desired output. Like regular escaping it can, however, become confusing when many quotes are used. For instance, the code to print the above source code is:
print "print ""Nancy said """"Hello World!"""" to the crowd."""
Configurable alternative quoting delimiters
[edit]In contrast to dual delimiters, multiple delimiters are even more flexible for avoiding delimiter collision.[7]: 63
For example, in Perl:
print qq^Nancy doesn't want to say "Hello World!" anymore.^;
print qq@Nancy doesn't want to say "Hello World!" anymore.@;
print qq(Nancy doesn't want to say "Hello World!" anymore.);
all produce the desired output through use of quote operators, which allow any convenient character to act as a delimiter. Although this method is more flexible, few languages support it. Perl and Ruby are two that do.[7]: 62 [28]
Content boundary
[edit]A content boundary is a special type of delimiter that is specifically designed to resist delimiter collision. It works by allowing the author to specify a sequence of characters that is guaranteed to always indicate a boundary between parts in a multi-part message, with no other possible interpretation.[29]
The delimiter is frequently generated from a random sequence of characters that is statistically improbable to occur in the content. This may be followed by an identifying mark such as a UUID, a timestamp, or some other distinguishing mark. Alternatively, the content may be scanned to guarantee that a delimiter does not appear in the text. This may allow the delimiter to be shorter or simpler, and increase the human readability of the document. (See e.g., MIME, Here documents).
Whitespace or indentation
[edit]Some programming and computer languages allow the use of whitespace delimiters or indentation as a means of specifying boundaries between independent regions in text.[30]
Regular expression syntax
[edit]In specifying a regular expression, alternate delimiters may also be used to simplify the syntax for match and substitution operations in Perl.[31]
For example, a simple match operation may be specified in Perl with the following syntax:
$string1 = 'Nancy said "Hello World!" to the crowd.'; # specify a target string
print $string1 =~ m/[aeiou]+/; # match one or more vowels
The syntax is flexible enough to specify match operations with alternate delimiters, making it easy to avoid delimiter collision:
$string1 = 'Nancy said "http://Hello/World.htm" is not a valid address.'; # target string
print $string1 =~ m@http://@; # match using alternate regular expression delimiter
print $string1 =~ m{http://}; # same as previous, but different delimiter
print $string1 =~ m!http://!; # same as previous, but different delimiter.
Here document
[edit]A here document allows the inclusion of arbitrary content by specifying a special end sequence. Many languages support this including PHP, Bourne shell, ruby and perl. A here document starts by describing what the end sequence is and continues until that sequence occurs at the start of a new line.[32] If the content is known, this technique avoids delimiter collision since an end sequence can be chosen that does not exist in the content.
An example in perl:
print <<ENDOFHEREDOC;
It's very hard to encode a string with "certain characters".
Newlines, commas, and other characters can cause delimiter collisions.
ENDOFHEREDOC
This code prints:
It's very hard to encode a string with "certain characters". Newlines, commas, and other characters can cause delimiter collisions.
ASCII armor
[edit]Although principally used as a mechanism for text encoding of binary data, ASCII armoring is a programming and systems administration technique that also helps avoid delimiter collision in some circumstances.[33][34] This technique is more complicated than many other collision avoidance techniques, and therefore is less suitable for small applications and simple data formats. The technique employs a special encoding scheme, such as base64, to ensure that delimiter or other significant characters do not appear in transmitted data. It prevents multilayered escaping, i.e. for double-quotes.
This technique is used, for example, in ASP.NET, and is closely associated with the VIEWSTATE component of that system.[35] This prevents delimiter collision and ensures that incompatible characters will not appear inside the HTML code, regardless of what characters appear in the original (decoded) text.[35] The following example demonstrates how this technique works.
The following code fragment shows an HTML tag in which the VIEWSTATE value contains double-quotes – characters that are incompatible with the delimiters of the HTML tag. The code is not valid and would fail.
<input type="hidden" name="__VIEWSTATE" value="BookTitle:Nancy doesn't say "Hello World!" anymore." />
To store arbitrary text in an HTML attribute, HTML entities can be used. In this case " stands for double-quote.
<input type="hidden" name="__VIEWSTATE" value="BookTitle:Nancy doesn't say "Hello World!" anymore." />
Alternatively, any encoding could be used that doesn't include characters that have special meaning in the context, such as base64:
<input type="hidden" name="__VIEWSTATE" value="Qm9va1RpdGxlOk5hbmN5IGRvZXNuJ3Qgc2F5ICJIZWxsbyBXb3JsZCEiIGFueW1vcmUu" />
Or percent-encoding:
<input type="hidden" name="__VIEWSTATE" value="BookTitle:Nancy%20doesn%27t%20say%20%22Hello%20World!%22%20anymore." />
Notes
[edit]- ↑ According to ANSI INCITS 4-1986 (R2007): "FILE SEPARATOR, GROUP SEPARATOR, RECORD SEPARATOR, and UNIT SEPARATOR are intended for applications in which the information separators are used hierarchically. The ascending order is then US, RS, GS, FS. In this case, data normally delimited by a particular separator cannot be split by a higher-order separator but will be considered as delimited by any higher-order separator."
- ↑ Later, the Unicode Consortium adopted ASCII's code points. Thus, the 8-bit ASCII-compatible UTF-8 encodes these control characters identically to ASCII. In the 16-bit UTF-16 and 32-bit UTF-32 encodings, their lowest-byte is identical to their corresponding ASCII byte and their upper-byte(s) are 0x00.
- ↑ These provide a visual representation of the corresponding control codes. Their code is different from their corresponding control character.
See also
[edit]- CDATA – Section delimiter in the markup languages SGML and XML
- Decimal separator – Numerical symbol
- Delimiter-separated values – Text format for indicating tabular data
- Escape sequence – Series of characters with a special meaning
- Newline – Special characters in computing signifying the end of a line of text
- String literal – Delimited series of characters that represent a string in code
- Tab-separated values – Text format for tabular data using a tab between fields
References
[edit]- ↑ "Definition: delimiter". Federal Standard 1037C - Telecommunications: Glossary of Telecommunication Terms. Archived from the original on 2013-03-05. Retrieved 2019-11-25.
- ↑ "What is a Delimiter?". www.computerhope.com. Archived from the original on 2020-11-16. Retrieved 2020-08-09.
- ↑ Rohl, Jeffrey S. (1973). Programming in Fortran. Oxford Oxfordshire: Oxford University Press. ISBN 978-0-7190-0555-8. describing the method in Hollerith notation under the Fortran programming language.
- 1 2 de Moor, Georges J. (1993). Progress in Standardization in Health Care Informatics. IOS Press. ISBN 90-5199-114-2. p. 141
- ↑ Friedl, Jeffrey E. F. (2002). Mastering Regular Expressions: Powerful Techniques for Perl and Other Tools. O'Reilly. ISBN 0-596-00289-0. p. 319
- ↑ Scott, Michael Lee (1999). Programming Language Pragmatics. Morgan Kaufmann. ISBN 1-55860-442-1.
- 1 2 3 Wall, Larry; Orwant, Jon (July 2000). Programming Perl (Third ed.). O'Reilly. ISBN 0-596-00027-8.
- ↑ Kaufmann, Matt (2000). Computer-Aided Reasoning: An Approach. Springer. ISBN 0-7923-7744-3.p. 3
- ↑ Meyer, Mark (2005). Explorations in Computer Science. Oxford Oxfordshire: Oxford University Press. ISBN 978-0-7637-3832-7. references C-style programming languages prominently featuring curly brackets and semicolons.
- ↑ Dilligan, Robert (1998). Computing in the Web Age. Oxford Oxfordshire: Oxford University Press. ISBN 978-0-306-45972-6.Describes syntax and delimiters used in HTML.
- 1 2 Schwartz, Randal (2005). Learning Perl. Oxford Oxfordshire: Oxford University Press. ISBN 978-0-596-10105-3.Describes string literals.
- ↑ Watt, Andrew (2003). Sams Teach Yourself Xml in 10 Minutes. Oxford Oxfordshire: Oxford University Press. ISBN 978-0-672-32471-0. Describes XML processing instruction. p. 21.
- ↑ Cabrera, Harold (2002). C# for Java Programmers. Oxford Oxfordshire: Oxford University Press. ISBN 978-1-931836-54-8. Describes single-line and multi-line comments. p. 72.
- ↑ "Jakarta Server Pages Specification, Version 4.0akarta Server Pages Specification, Version 4.0". GitHub. Archived from the original on 2023-02-10. Retrieved 2023-02-10.
- ↑ Information processing — ISO 7-bit coded character set for information interchange (2 ed.). ISO. 1983.
- ↑ ANSI INCITS 4-1986 (R2007): American National Standard for Information Systems – Coded Character Sets – 7-Bit American National Standard Code for Information Interchange (7-Bit ASCII), 2007 [1986]
- 1 2 SixArm/usv, SixArm, 2026-06-07, retrieved 2026-08-31
- ↑ Cerf, V. G. (October 1969). ASCII format for network interchange (Report). Internet Engineering Task Force. doi:10.17487/RFC20.
- ↑ "ANSI/NBS-ICST 1-1986: American National Standard for information systems - fingerprint identification - data format for information interchange" (PDF). American National Standards Institute (NIST). 1986-08-25. p. 4-5. Retrieved 2026-08-31.
{{cite web}}: CS1 maint: url-status (link) - ↑ Mirzaab. "GS1 bar codes and QR codes - Supply Chain Management | Dynamics 365". learn.microsoft.com. Retrieved 2026-08-30.
- ↑ "GS1 General Specifications Standard: Release 26.0, Ratified, Jan 26". 2026.
- ↑ Williams, N. (February 2015). JavaScript Object Notation (JSON) Text Sequences (Report). Internet Engineering Task Force. doi:10.17487/RFC7464.
- ↑ Henderson, Joel Parker (2024-03-16). Unicode Separated Values (USV) (Report). Internet Engineering Task Force.
- ↑ "usv - Rust". docs.rs. Retrieved 2026-08-31.
- ↑ Friedl, Jeffrey (2006). Mastering Regular Expressions. Oxford Oxfordshire: Oxford University Press. ISBN 978-0-596-52812-6. describing solutions for embedded-delimiter problems p. 472.
- ↑ Kahrel, Peter (2006). Automating InDesign with Regular Expressions. O'Reilly. p. 11. ISBN 0-596-52937-6.
- ↑ "Quotation marks in string expressions". learn.microsoft.com. Retrieved 2026-09-02.
- ↑ Yukihiro, Matsumoto (2001). Ruby in a Nutshell. O'Reilly. ISBN 0-596-00214-9. In Ruby, these are indicated as general delimited strings. p. 11
- ↑ Network Protocols Handbook. Javvin Technologies Inc. 2005. ISBN 0-9740945-2-8. p. 26
- ↑ Computational Linguistics and Intelligent Text Processing. Oxford Oxfordshire: Oxford University Press. 2001. ISBN 978-3-540-41687-6. Describes whitespace delimiters. p. 258.
- ↑ Friedl, Jeffrey (2006). Mastering Regular Expressions. Oxford Oxfordshire: Oxford University Press. ISBN 978-0-596-52812-6. page 472.
- ↑ "Perl operators and precedence". Archived from the original on 2012-07-17. Retrieved 2011-11-11.
- ↑ Rhee, Man (2003). Internet Security: Cryptographic Principles, Algorithms and Protocols. John Wiley and Sons. ISBN 0-470-85285-2.(an example usage of ASCII armoring in encryption applications)
- ↑ Gross, Christian (2005). Open Source for Windows Administrators. Charles River Media. ISBN 1-58450-347-5.(an example usage of ASCII armoring in encryption applications)
- 1 2 Kalani, Amit (2004). Developing and Implementing Web Applications with Visual C# . NET and Visual Studio . NET. Que. ISBN 0-7897-2901-6.(describes the use of Base64 encoding and VIEWSTATE inside HTML source code)