10 commits

Author	SHA1	Message	Date
janis	bf9d07b462	init tokeniser with buffer	2025-10-29 20:39:22 +01:00
janis	39e8d6ae96	move defintions out of tokeniser into include file	2025-10-29 16:21:01 +01:00
janis	86bbab90c3	modularise test with shared rust structs ... add expect/unwrap token methods to tokeniser to aid with parsing	2025-10-29 14:00:17 +01:00
janis	5f63d4303e	formatting, strings, comments	2025-10-28 18:16:39 +01:00
janis	43a06ad55c	test numbers	2025-10-28 17:16:28 +01:00
janis	44ee13246e	is_number	2025-10-28 17:07:05 +01:00
janis	1886231dcd	return lexeme str slice from find_lexeme function	2025-10-28 16:30:15 +01:00
janis	cfc1c8e3b3	more tests for tokeniser	2025-10-28 13:12:27 +01:00
janis	63fbca378b	can tokenise simple tokens	2025-10-28 12:44:08 +01:00
janis	719451b935	reduce warnings, enable edition 2024, fix tokeniser (1/?)	2025-10-28 12:06:24 +01:00