14 commits

Author	SHA1	Message	Date
janis	838c96f04f	update python script, add slash and star tokens	2025-10-30 00:11:46 +01:00
janis	8740fea99d	fix tokeniser for: void	2025-10-29 22:39:27 +01:00
janis	bf9d07b462	init tokeniser with buffer	2025-10-29 20:39:22 +01:00
janis	62751f30ab	move more stuff to shared test file	2025-10-29 16:20:42 +01:00
janis	86bbab90c3	modularise test with shared rust structs ... add expect/unwrap token methods to tokeniser to aid with parsing	2025-10-29 14:00:17 +01:00
janis	6a6af53667	bump allocator	2025-10-28 20:40:25 +01:00
janis	5f63d4303e	formatting, strings, comments	2025-10-28 18:16:39 +01:00
janis	43a06ad55c	test numbers	2025-10-28 17:16:28 +01:00
janis	82104cfe0d	edge case identifiers, simple function parsing	2025-10-28 16:52:17 +01:00
janis	26c37e9dd3	test identifiers	2025-10-28 16:48:13 +01:00
janis	1886231dcd	return lexeme str slice from find_lexeme function	2025-10-28 16:30:15 +01:00
janis	cfc1c8e3b3	more tests for tokeniser	2025-10-28 13:12:27 +01:00
janis	63fbca378b	can tokenise simple tokens	2025-10-28 12:44:08 +01:00
janis	719451b935	reduce warnings, enable edition 2024, fix tokeniser (1/?)	2025-10-28 12:06:24 +01:00