CSV — Shift-JIS Japanese
A Shift-JIS encoded Japanese CSV — for CJK charset detection and Shift-JIS→UTF-8 conversion in importers.
id,name,city
1,���Y,����
2,�Ԏq,���
Specifications
- Encoding
- Shift-JIS
- Language
- Japanese
- Rows
- 2
Testing contract
Expected to pass- Scenario
- Exercise CSV — Shift-JIS Japanese in its encodings workflow. A Shift-JIS encoded Japanese CSV — for CJK charset detection and Shift-JIS→UTF-8 conversion in importers.
- Expected result
- 2 data records using ',' delimiters and single-byte preserved octets; header fields are id, name, city; data-record widths (columns:count) are {"3":2}. Declared feature checks: language=Japanese.
What is a .csv file?
CSV (Comma-Separated Values) is a plain-text tabular format where rows are lines and fields are separated by commas, with quoting rules for values that contain delimiters, quotes, or newlines. It has no formal type system and depends on encoding and dialect conventions. It is the most portable format for tabular data exchange.
How to use this file
Use an example CSV to test parsers against quoting and embedded-delimiter edge cases, header handling, encoding detection, and import pipelines into databases or spreadsheets.
How to use this file for testing
“CSV — Shift-JIS Japanese” is a deterministic Novus Examples fixture for CSV parsing, Encoding detection, Internationalization. Clean and deliberately messy CSVs, quoted commas, embedded newlines, ragged rows, odd delimiters, and encodings.
Documented properties for this file: 2 rows · Japanese · Shift-JIS. Compare results against paired or grouped companions on this page when present (clean↔damaged, searchable↔scanned, or format twins) so scores stay reproducible across runs.
Download the file once, keep the path stable in CI or local scripts, and treat the spec table as the contract: dimensions, seeds, field lists, and roles are intentional. Corrupt or invalid samples are labelled as such, expect parsers to fail loudly rather than silently accept them.
Data fixtures document their exact quirks (delimiters, encodings, null handling, schema, and row counts) in the spec table. Point your parser or importer at the file and assert it handles the documented edge cases; clean and deliberately-messy siblings make before/after diffs straightforward.
Feed the file to your parser and assert it handles the documented quirks, quoted delimiters, embedded newlines, ragged rows, or invalid syntax; the valid↔invalid distinction is labelled in the title.
Code examples
import pandas as pd
df = pd.read_csv("shift-jis.csv")
print(df.head())
print(df.dtypes)Related files
- csvLocale Currency Formats — eleven locales (CSV)12345.60 in each locale's own currency, with the fraction-digit count and the symbol placement that produced it. JPY takes zero decimals and rounds; six rows separate the amount from the symbol with U+00A0 rather than a plain space; and ar-EG puts the amount in Arabic-Indic digits.
- csvLocale Number Formats — twelve locales (CSV)The same number, 1234567.89, rendered for twelve locales, with the group and decimal separator that produced each one named by code point and present literally in the cell. Four of the separators are not ASCII — U+00A0, U+202F, U+2019 and U+066C — and ar-EG additionally switches the digits themselves.
- csvCSV — ISO-8859-1 (Latin-1)A Latin-1 CSV with high-bit accented characters — for legacy 8-bit charset detection.
- csvCSV — Mixed CRLF and LFA CSV that mixes CRLF and LF row endings within one file — newline normalisation fixture.
- csvCSV — UTF-16 BE with BOMUTF-16 big-endian CSV with BOM — for byte-order detection in tabular importers.
- csvCSV — UTF-16 LE with BOMThe same three data rows as UTF-16 little-endian with a BOM — wide-character CSV decoding fixture.
Generated by generation/data_encodings_wave_c.py. Free for any use, no attribution required, license.