Re: [Tinycc-devel] Two compilers at once

[uso ewin]

On Fri, Jun 21, 2019 at 4:58 AM Karl Yerkes <address@hidden> wrote:
>
>
> Steve Fan --
>
> On Thu, Jun 20, 2019 at 7:14 PM Steve Fan <address@hidden> wrote:
>>
>> I’m sorry to intervene in the matter, but as you look at the design of TCC, 
>> you will see that it is not designed for re-entrancy.
>
>
> yes. i understand. i am not trying to make it reentrant. i want to 
> hot-recompile-swap a function with very low latency and i am willing to 
> settle for a hack.
>
>
>> Foolish younger me, I used to have a hardworking re-entrant fork of TCC 
>> working but the one of the project owner denounced my idea, calling me to 
>> adapt TLS for each and every global variables instead of rewriting most of 
>> the TCC functions to add TCCState for them.
>
>
> good for you.. and that sucks.
>
>
>> TCC is just a hobbyist bone picked from OTCC by the legend Fabrice Bellard 
>> himself, but he has abandoned his child, TCC, ever since.
>
>
> yes. i get that.
>
>
>> I used to be like you, I want to research on TCC, but now I do warn you to 
>> not touch this egregious code again.
>
>
> i am not aiming to change the TCC source code. i actually only want to hack 
> around what currently exists. that said, i might end up helping out somehow.
>
>
>> Go figure out your own parser. Maybe learn Bison or ANTLR for a basic 
>> parser, then read some paper about register allocation (which is a very 
>> bitchy problem for it was actually NP-Complete), read the specs for ELF, and 
>> read the respective assembly handbook for each and every codegen backend. 
>> You can reuse the ELF/PE code from TCC I reckon, but
>
>
> thank you. this is good advice and good info. i know parsing and grammars. i 
> write my grammars/parsers in perl6 :)
>
> maybe generating x86, ARM, or LLVM-IR is not that hard and i should go do 
> that. (right now it is black magic to me.)
>
> but my aim is actually to hack something together to further my aims in a 
> different domain: sound.
>
> i am quite close with TCC.
>
>
>>
>> just don’t try to make anything happen to TCC, a dying project.
>
>
> i hear your warning. you may very well be right, but to me TCC seems... 
> mature, not dying. maybe it's not going to be much more than what it is, but 
> i bet it'll hold its value for a while.
>
> -- karl
>
>>
>>
>>
>> From: Karl Yerkes
>> Sent: Friday, 21 June 2019 9:24 AM
>> To: address@hidden
>> Subject: Re: [Tinycc-devel] Two compilers at once
>>
>>
>>
>> thank you for your replies. your help and insights are much appreciated.
>>
>>
>>
>> I'm loving TCC!
>>
>>
>>
>> however, i've reached the limit of my current hot-swap function hack that 
>> uses two state-commingling TCC instances. after a while, the instances stop 
>> working properly and I get strange compile errors.
>>
>>
>>
>> so, i am interested in executing some hack to allow multiple independent TCC 
>> instances in the same application. i am certain that "the old trick of just 
>> opening a pipe, forking and
>>
>> then sending back the result" will not work for my purposes. if i can't get 
>> one of these hacks working, i have to go look at LLVM and cling again :/
>>
>>
>>
>> as i understand it, option 2 (from Giovanni Mascellani) is using libtcc to 
>> compile libtcc from source into memory. that sounds awesome. i spent most of 
>> today trying this on windows.. and also learning to compile TCC on windows. 
>> here's my attempt:
>>
>>
>>
>> // boot.cpp
>> //
>> // tcc version 0.9.27 (x86_64 Windows)
>> // install: c:/tcc
>> // include:
>> //   c:/tcc/include
>> //   c:/tcc/include/winapi
>> // libraries:
>> //   c:/tcc/lib
>> //   C:/Windows/system32
>> // libtcc1:
>> //   c:/tcc/lib/libtcc1-64.a
>> //
>> // build and run with:
>> //   cd tinycc
>> //   cl /MD boot.cpp c:\tcc\libtcc64.lib
>> //   boot.exe
>> //
>> //  fails like this:
>> //  need 457827 bytes
>> //  relocating to 21f0a402fd0
>> //  tcc: error: library 'libtcc1-64.a' not found
>> //  Assertion failed: dec != nullptr, file boot.cpp, line 69
>> //
>> #include "libtcc.h"
>>
>> #include <cassert>
>> #include <iostream>
>>
>> int main() {
>>   TCCState* instance = tcc_new();
>>
>>   // i'm on Windows 10 x64. what else do i need here?
>>   tcc_set_options(instance, "-shared");
>>   tcc_define_symbol(instance, "TCC_TARGET_X86_64", "");
>>   tcc_define_symbol(instance, "TCC_TARGET_PE", "");
>>   tcc_define_symbol(instance, "LIBTCC_AS_DLL", "");
>>
>>   // i needed this define on Windows to get tcc_relocate in the DLL
>>   tcc_define_symbol(instance, "TCC_IS_NATIVE", "");
>>
>>   tcc_set_output_type(instance, TCC_OUTPUT_MEMORY);
>>
>>   // maybe add more source files?
>>   tcc_add_file(instance, "libtcc.c");
>>
>>   int size = tcc_relocate(instance, (void*)0);
>>   void* memory = malloc(size);
>>
>>   printf("need %d bytes\n", size);
>>   printf("relocating to %llx\n", (unsigned long long int)memory);
>>   tcc_relocate(instance, memory);
>>
>>   // TCCState *tcc_new(void);
>>   using New = TCCState* (*)(void);
>>   New _new = (New)tcc_get_symbol(instance, "tcc_new");
>>   assert(_new != nullptr);
>>
>>   // int tcc_set_output_type(TCCState *s, int output_type);
>>   using SetOutputType = int (*)(TCCState*, int);
>>   SetOutputType _set_output_type =
>>       (SetOutputType)tcc_get_symbol(instance, "tcc_set_output_type");
>>   assert(_set_output_type != nullptr);
>>
>>   // int tcc_compile_string(TCCState *s, const char *buf);
>>   using CompileString = int (*)(TCCState*, const char*);
>>   CompileString _compile_string =
>>       (CompileString)tcc_get_symbol(instance, "tcc_compile_string");
>>   assert(_compile_string != nullptr);
>>
>>   // int tcc_relocate(TCCState *s1, void *ptr);
>>   using Relocate = int (*)(TCCState*, void*);
>>   Relocate _relocate = (Relocate)tcc_get_symbol(instance, "tcc_relocate");
>>   assert(_relocate != nullptr);
>>
>>   // void *tcc_get_symbol(TCCState *s, const char *name);
>>   using GetSymbol = void* (*)(TCCState*, const char*);
>>   GetSymbol _get_symbol = (GetSymbol)tcc_get_symbol(instance, 
>> "tcc_get_symbol");
>>   assert(_get_symbol != nullptr);
>>
>>   // can i delete the instance now??
>>   // my experiments suggest no.
>>   // tcc_delete(instance);
>>
>>   // use the new, relocated instance to compile something
>>   TCCState* born_in_memory = _new();
>>   assert(born_in_memory != nullptr);
>>   _set_output_type(born_in_memory, TCC_OUTPUT_MEMORY);
>>   _compile_string(born_in_memory, "int dec(int t) { return t - 1; }");
>>   _relocate(born_in_memory, TCC_RELOCATE_AUTO);
>>   using Foo = int (*)(int);
>>   Foo dec = (Foo)_get_symbol(born_in_memory, "dec");
>>   assert(dec != nullptr);
>>   printf("dec(2) == %d\b", dec(2));
>> }
>>
>>
>>
>> am i on the right track here? any suggestions would help a lot. i think i 
>> could add the path to libtcc1-64.a, but i would rather compile all the .c 
>> files i need so there's no need to find libtcc1-64.a at runtime.
>>
>>
>>
>> -
>>
>>
>>
>> option 1 (from Christian Jullien) is something that i don't quite 
>> understand. i am just failing to connect the dots given i the pseudocode:
>>
>>
>>
>> // tcc1.cpp:
>> namespace tcc1 {
>> #include "libtcc.h"
>>
>>   // does this mean i should compile libtcc as c++ (not c) under the tcc1 
>> namespace?
>> };
>>
>> // tcc2.cpp:
>> namespace tcc2 {
>> #include "libtcc.h"
>>
>>    // does this mean i should compile libtcc as c++ (not c) AGAIN under the 
>> tcc2 namespace?
>>
>>  // OR does this mean i should write a wrapper interface for libtcc here?
>> };
>>
>> // main.cpp:
>> int main() {
>>   tcc1::TCCState* a = nullptr;
>>   char (*A)(int) = nullptr;
>>   assert((a = tcc1::tcc_new()) != nullptr);
>>   tcc2::TCCState* b =
>>       tcc2::tcc_new();  // remove this line to make the program work
>> }
>>
>> also "But it may not be so simple. For some projects it worked flawlessly 
>> and failed for some others" is mysterious to me. my apologies for being too 
>> ignorant to put it together! a little more help on this option might help me 
>> a lot.
>>
>>
>>
>> sorry for the long email.
>>
>>
>>
>> -- karl
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Sat, May 25, 2019 at 12:30 AM Giovanni Mascellani <address@hidden> wrote:
>>
>> Hi,
>>
>> Il 25/05/19 06:24, Christian Jullien ha scritto:
>> > Because Karl uses C++11, it may be /theoretically/ possible for him to
>> > load two different tcc instances having different C++ namespaces and
>> > compiled in two distinct translation units.
>>
>> If you don't mind some quite dirty hacking (and if you mind it you
>> shouldn't probably use tcc), I also believe that you can have two
>> instances in the same address space by using the first one to compile
>> another one (or many others) and relocating them to different addresses.
>> Then each of them is a completely independent compiler with independent
>> state. Of course this means that you need tcc's source code at runtime.
>> Otherwise there is the old trick of just opening a pipe, forking and
>> then sending back the result.
>>
>> Have fun, Giovanni.
>> --
>> Giovanni Mascellani <address@hidden>
>> Postdoc researcher - Université Libre de Bruxelles
>>
>> _______________________________________________
>> Tinycc-devel mailing list
>> address@hidden
>> https://lists.nongnu.org/mailman/listinfo/tinycc-devel
>>
>>
>>
>> _______________________________________________
>> Tinycc-devel mailing list
>> address@hidden
>> https://lists.nongnu.org/mailman/listinfo/tinycc-devel
>
> _______________________________________________
> Tinycc-devel mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/tinycc-devel

Stupide question, but can't you just use 2 instances of tcc in 2
diferent process ?
Maybe a simpler hack, instead of making it reentrent would be to add
optionaly _Thread_local
attribute to tcc globals and execute the 2 instances in 2 diferent threads.

Matthias,