Makefile: Overlapping includes across modules, and module cache files

Authoring TADS
1

Okay, I have a rather complex problem, and I’m not understanding the docs enough to see how this would work.

Related Question 1:

I have a TADS folder for my projects; it’s called tads3projects. In that folder are individual folders for games and modules. Modules are collections of source files, united with a .h file, which can be pulled into any game project through the game’s individual makefile.

For example:

tads3projects/
    InventorCore/ (Module)
        inventorCore.h
    QuickMaffs/ (Module)
        quickMaffs.h
    MyGameProject/ (Game)
        gameSourceFile.t
        Makefile.t3m

I have heard that overlapping include lines can be handled by using the -source argument in the makefile. If inventorCore.h and quickMaffs.h both require #include <file.h> to function, then how would I add these two modules into my game with the game’s makefile without needing to omit <file.h> from both modules, and remembering to always add <file.h> to every future game project?

Related Question 2:

If a module has cache files that need to be opened and processed during preinit, how would I do this without needing to copy the cache files in every future game project?

A lot of this can probably be solved with symlinks, but I would rather not need to show my user folder’s structure in public GitHub commits, and also have to carefully manage symlinks, especially if the makefile can handle this. I had to manage symlinks in the early version of my I Am Prey repo, and I kept having issues with how git handled it, and it became too annoying to manage, so I just hard-copied my modules into the game folder.

I’m now trying to break stuff back out and create a more manageable module library with the makefile, but admittedly every time I think I find new guidance about the makefile, t3make doesn’t seem to understand me, and the docs don’t clarify things very well.

I’m requesting patience, because I’ll be asking a lot of follow-up questions, as I try to get stuff working based on any answers/help provided. Whatever gets fully nailed down will be the enduring foundation of my module library going forward, so I’m trying to get a single system properly figured out.

Thank you for your time.

2 Likes
2

The way I handle “cascading” module dependencies is more or less:

  • Include a header define in all module header files. So for a project fooBarBaz, fooBarBaz.h will include a line like #define FOO_BAR_BAZ_H (screaming snake case is just a convention, other conventions work just as well)
  • In everything that depends on fooBarBaz, their header file gets something like:
                #include "fooBarBaz.h"
                #ifndef FOO_BAR_BAZ_H
                #error "This module requires the fooBarBaz module."
                #endif // FOO_BAR_BAZ_H
  • At least one *.t file in each project has to #include the project’s own header file
  • Each module gets a *.tl library file
  • Modules get added via -lib [path to module .tl]/[name of .tl file]. So if fooBarBaz’s library file is fooBarBaz.tl and the module is in its own fooBarBaz directory, then something like -lib ../fooBarBaz/fooBarBaz (or, equivalently, -lib ../fooBarBaz/fooBarBaz.tl, but adding the .tl is not necessary)

In general you don’t have to #include module/library header files in every source file of a project that’s using the module, you just have to do that if you reference something in the header file itself (like a template statement or something like that).

And you generally shouldn’t be using #include to import dependencies directly, which is how TADS2 did things but is deprecated in TADS3. Instead create a library file for modules and then point to it via -lib in the makefile of anything that needs to include it.

It’s a bit of an irritation because you end up having to include all the modules’ dependencies in a project’s makefile (instead of them getting automagically added by the modules themselves). But this is usually something you only have to worry about once per project, so it’s not that big of a deal.

1 Like
3

Okay so if I point to a module’s *.tl file in the makefile, do I need to put an include for the module’s *.h file in a game source file still? Or does pointing to the *.tl file import all the connected module sources files already?

I still don’t understand how a *.tl file works, how to make one, or how to use them. Do you know which book of the docs that’s in? Because I’m learning that a lot of the documentation for t3make is hidden behind specific chains of keyword hyperlinks if I’m trying to find them from the Adv3Lite docs.

4

Only if the code in the individual source file refers to something that’s only in the header. So things like object template declarations, macros, and things like that. If you’re just referring to classes, functions, or whatever that are provided by the module/library, you don’t necessarily need to include the module-specific header file.

In a lot of cases you could in theory get away with not even having a module-specific header file. I always have one, but that’s because I use a lot of modules and I pretty much always include a module-specific #define just to make it easier to do dependency tracking.

If you want a complete example, you can check out the simpleGraph module. Here we’ll just look at the module’s header file (simpleGraph.h) and library file (simpleGraph.tl). First the header:

//
// simpleGraph.h
//

// Include Dijkstra pathfinding logic
#define SIMPLE_GRAPH_DIJKSTRA

// Enable caching of pathfinding data
//#define SIMPLEGRAPH_DIJKSTRA_CACHE

// Include logic for computing the Laplacian matrix for the graph
//#define SIMPLE_GRAPH_LAPLACIAN

// Include logic for computing the Cheeger constant for the graph
//#define SIMPLE_GRAPH_CHEEGER

SimpleGraph template 'id'?;
SimpleGraphVertex template 'id'?;
SimpleGraphEdge template 'id0' 'id1';


// Don't comment this out.  It's how we announce to the compiler that we're
// here.  This is used for dependency checking in other modules.
#define SIMPLE_GRAPH_H

We can ignore most of the #define statements, which just control which features of the module are enabled. These only need to be “visible” to module’s code; your game/project code probably won’t include anything that refers to the module’s compile-time options.

There’s the #define SIMPLE_GRAPH_H line at the very bottom which is used for dependency checking, but that’s just me being me, it’s not required by TADS3. So ignore that.

All that’s left is the template lines in the middle. These are just to allow game code to contain declared-in-source graphs, vertices, and edges. So instead of having to do something like this in a method somewhere:

        // With no template you have to do
        local obj = new SimpleGraph();
        obj.addVertex('foo');
        obj.addVertex('bar');
        obj.addEdge('foo', 'bar');

…you can do something like…

        // With a template you can
        myGraph: SimpleGraph 'someGraphID';
        +SimpleGraphVertex 'foo';
        +SimpleGraphVertex 'bar';
        +SimpleGraphEdge 'foo' 'bar';

…similar to the way you declare rooms, actors, and so on.

The point to this being that the class declarations will be available to your game code even if you didn’t put a #include "simpleGraph.h" in the game source file referring to the SimpleGraph class. But the template wouldn’t be. So the first example above would work fine without including the header, but the second would not.

The library file is just a list of the source files in the module, along with a name. Here’s simpleGraph.tl:

name: Simple Graph Library
source: simpleGraphModule.t
source: simpleGraph.t
source: simpleGraphDirected.t
source: simpleGraphPreinit.t
source: simpleGraphDijkstra.t
source: simpleGraphEdge.t
source: simpleGraphVertex.t
source: simpleGraphLaplacian.t
source: simpleGraphCheeger.t
source: simpleGraphNextHopCache.t
source: simpleGraphSubgraph.t

When you add -lib [path to module]/simpleGraph.tl to a makefile, that tells t3make to build all of the source files listed in the *.tl and to include the resulting object files when compiling the project. That makes all the classes, objects, functions, and so on that are defined in the listed source files available to code in the project that imported them.

But even if each file in the module includes its own header file (and in this case they do) the “stuff” in the header still won’t be available to anything in your project unless you include the header file yourself.

Hope that all makes sense.

As for where it’s documented…not sure. It might be in the documentation about separate compilation (which is the search term if you’re looking for stuff on the T3-specific module/library stuff), but I think my knowledge on the subject is a combination of the documentation, fiddling with existing examples, trial and error, and source diving.

1 Like
5

Okay so if I have a class called Foo in FooBar.t, then I just need to point to FooBar.t in the *.tl file, and point to the *.tl file in the makefile. From there I can do…

baz: Foo {
    // Stuff
}

…in the game code, and it all compiles fine?

But if I have stuff declared in the actual header file, then the header needs to be included…?

Is there a reason why I shouldn’t put the header contents into a *.t file, and point to it with the module’s *.tl file, which avoids the awkward situation of the *.h file?

6

Things like #defines and template statements aren’t going to behave the way you want them to if you put them in a module source file instead of a header file.

Here’s a minimalist module source, call it fooBarBaz.t:

#charset "us-ascii"
#include <adv3.h>
#include <en_us.h>

class FooBarBaz: object
        foozle = '[This space intentionally left blank]'
;

// Don't put this in a souce file!
FooBarBaz template 'foozle'?;

…and a library file for it, fooBarBaz.tl:

name: fooBarBaz Library
source: fooBarBaz.t

Then we have a game, sample.t:

#charset "us-ascii"
#include <adv3.h>
#include <en_us.h>

myFoo: FooBarBaz 'Some alternate text.';

versionInfo: GameID;
gameMain: GameMainDef
        newGame() {
                local obj;

                obj = new FooBarBaz;
                "<<toString(obj.foozle)>>\n ";
        }
;

…that we (try to) compile with the makefile:

-D LANGUAGE=en_us
-D MESSAGESTYLE=neu
-Fy obj -Fo obj
-o game.t3
-lib system
-lib adv3/adv3
-lib fooBarBaz
-source sample

…but we get:

sample.t(5): error: object definition does not match any template

…because the template statement isn’t an object/symbol/whatever that can be “seen” from the code in sample.t.

Remove the myFoo: FooBarBaz 'Some alternate text.'; line from sample.t and it’ll compile fine, and when you run the “game”, you’ll get:

[This space intentionally left blank]

…which means that the sample.t can use the FooBarBaz class, just not the template. Similarly, if we add a line:

#define FOO_BAR_BAZ_H

…to fooBarBaz.t and then re-write sample.t as:

#charset "us-ascii"
#include <adv3.h>
#include <en_us.h>

//myFoo: FooBarBaz 'Some alternate text.';

versionInfo: GameID;
gameMain: GameMainDef
        newGame() {
                local obj;

                obj = new FooBarBaz;
                "<<toString(obj.foozle)>>\n ";

#ifdef FOO_BAR_BAZ_H
                "Foo, bar, and baz!\n ";
#endif // FOO_BAR_BAZ_H
        }
;

…the output will still be (only):

[This space intentionally left blank]

…because the code in sample.t can’t “see” the #define in fooBarBaz.t.

1 Like
7

…oh.

So only functions, classes, and objects work in a *.t file, and anything else needs to be directly imported from a *.h file?

8

I think #define statements/macros and template statements are the only things that need to be in a header file (if external callers need to be able use them).

I think everything else can be declared in a source file in a module, including things like enums, grammar rules, and so on.

Or at least I think both of those statements are true for everything I’ve personally encountered. You could dig through the compiler source to identify what’s just compile-time stuff and what’s exported in the symbol/object files but I don’t know of any place in the documentation that lists everything that is or is not included in either category.

1 Like
9

Thank you for the help.

It seems I have a lot of module restructuring to do. I’ve only been learning about how t3make and makefiles work through searching the forum. I have a lot of random blind spots.

10

If it makes you feel any better, I always feel the same way as well.

Like if you wanted to come up with a list of statements you could use for cold reading programmers, “You have the sneaking suspicion that you need to completely re-organize that code you’re working on” would be right at the top of the list.

2 Likes
11

Oh, also:

If I’m importing a module as a *.tl library file, and also including an *.h file, should I remove all the #include statements from the *.h file, because those will be handled by the *.tl file? Also, do I need to declare the *.h file in the makefile, or will the compiler know how to find the *.h file based on the library file’s path? Like, can I do #include <fooBar.h> if I have -lib fooBar in the makefile?

12

There shouldn’t be any conflicts there because you shouldn’t be using #include for *.t files, just *.h files. And you shouldn’t be putting *.h files in the *.tl file, just *.t files (or add them via -source in the makefile if the code that needs to “include” a *.t isn’t a module/library).

1 Like
13

Elements of this are covered in the TADS 3 (as opposed to adv3Lite specific) documentation. One place to start may be the article on Understanding Separate Compilation in the the TADS 3 Technical Manual (I don’t link to this from the adv3Lite Bookshelf as it’s mostly adv3 specific, so you’d need to go to it via the TADS 3 Bookshelf).

The chapter on Compiling and Linking at the start of Section II of the TADS 3 System Manual should tell you quite a bit of what you want to know. It may also be worth looking at the chapters on Source Code Structure and The Preprocessor in Section III.

1 Like
14

Ohhhhhh so that’s where everyone is learning this! Excellent! I think I might have the TADS 3 docs buried somewhere, so I’m absolutely checking those out next! Thanks for the pointer!

Noted!! I’ll be checking these out as well! Thanks again…! :grin:

15

Have you used a language with separate compilation before? I didn’t want to be explaining stuff that’s already obvious and I don’t really know TADS. But it feels like the basic distinction between header and source isn’t discussed in this thread, so…

In languages like C and C++ (and TADS 3, I guess), there are statements that generate data or executable code that goes in the output, and there are statements that just add temporary things that the compiler knows while it’s working.

So something like #define THREE 3 doesn’t generate any output: it just tells the compiler, “when you’re reading the source code and you see the text THREE, replace it with the text 3 before you continue compiling.” But the #define by itself doesn’t produce any compiled output, so if you put it in a .t file it won’t be available to other .t files that are compiled separately.

But something like the following generates an object as output.

startRoom: Room
	roomName = 'Starting Room'
	desc = "This is the starting room. "
;

And then #include just acts as if you’d deleted the #include statement and replaced it with the contents of the other file. So if you put your startRoom into a header file and include it in two different .t files… those will each get compiled separately, each will have a startRoom object in their output, and then when the linker tries to put the compiled files together into a single game it’ll go “wait, there are two objects named startRoom, how do I know which one is the real startRoom?” and it’ll be an error.

So the basic idea is that if it’s just an instruction to the compiler of how to generate code (the docs say this is just #define and templates for TADS), then you probably want to put in in a header file so you can #include it in multiple places. If you put it in a source file then it’ll only be available in that source file and the compiler won’t know about it when it’s compiling other source files (and sometimes you want it to be private like that). But if it’s a statement that generates code or data, then you want to put it in a source file so you only have one copy of that output, otherwise the linker will get confused when it goes to put all the compiled output files together into a game.

One other wrinkle: I don’t know how TADS works, but in most languages it’s an error to #define something twice (they’re constants, not variables) so it’s normal for header files to have guard statements to make sure they’re only included once for each run of the compiler (each source file). Kind of the flip side of what jbg had in post 2: foo.h would have an #ifndef right from the start to end of the file:

#ifndef FOO_H
#define FOO_H

// ... all the actual content

#endif // FOO_H

This is only really a problem if you have headers that include other headers: you wouldn’t say #include "foo.h" twice in a row in the same file, but if you have a useful_stuff.h that’s included by a bunch of other .h files… but it’s a good habit to put the guards in everywhere anyway.

Again, sorry if this is already completely obvious but I couldn’t tell for sure from what you’ve written here… (edit: also my brother was asking C++ questions yesterday so now I’m all like, “huh, that actually is a really weird arbitrary distinction because it used to be a separate text-replacement engine that ran before the compiler pass, I wonder if people know how this works?”)

2 Likes
16

Okay so that’s the wild thing, right? I’m largely self-taught in a lot of the languages that I use, but in all the resources I went through to learn C and C++ (including two university courses that required me to use it semi-proficiently), this was never brought up before, so I’m just now learning from your post that this is a Thing That Happens in multiple programming languages, including ones I have used before (C and C++).

1 Like
17

If you’re interested in more of the theory and design stuff about how compilers work, you could do worse than picking up a used copy of the Dragon Book. It’s pretty crunchy, but it’s the canonical reference on the subject. I suggest picking up a used copy because it’s a frequently-used textbook, and so new copies are expensive but there are a bunch of used copies out there. I think you can also just “check out” a copy from archive.org (like a e-library book) if you have an account, and accounts are free.

And in terms of general programming theory you might be interested in the Wizard Book, although it’s even more crunchy than the Dragon Book. It’s another one you should be able to find plenty of used copies of, but the contents are licensed under CC [some version] so you can get a “legit” PDF of the text for free.

2 Likes
18

I think using header guards like this usually isn’t necessary in T3 because T3 provides a C-ish #pragma once that accomplishes the same thing (in a slightly less verbose fasion). And T3 enforces a (sometimes slightly irritating) rule that filenames must be globally unique (in the project being compiled). So it’s unlikely for there to be any accidental collisions between different revs of the same header (the main “gotcha” for #pragma once in C).

There’s also a #pragma all_once that treats all headers as if they contain #pragma once, which resolves most issues surrounding headers-with-#include. As far as I know.

The reason I’ve been using header guard-ish #defines is because although it’s easy enough to get the compiler to keep track of what headers have already been #include-ed, as far as I know there’s no way to implement automagic checks/conditional #includes based on the compiler’s state.

3 Likes
19

Wait, really? Will it throw an error if this isn’t done? Because I swear after reorganizing I Am Prey, I have at least two source files in different folders but the source files share a name. Wondering if I’m breaking something without realizing it.

20

After writing a test case: it throws an error for source file name collisions (that is, you can’t have a file named foo.t in your game if there’s a foo.t in a module you’re using). Which is the slightly annoying thing I’ve encountered before.

But I have to revise my earlier statement because it doesn’t check for uniqueness of header filenames; it just (silently) uses the first one in the include path. So if you have a foo.h in your project and there’s a foo.h provided by a module, you’ll end up with the one from your project. If two different modules provide a foo.h, you’ll get whichever one was added via -lib first.

If you have multiple header files with the same name you can include them all by giving the full or relative path to the file, but in that case #pragma once/#pragma all_once won’t prevent errors if they try to #define the same things.

Interestingly, experimenting with this lead me to discover that the compiler doesn’t complain if you re-define enum tokens, individually or in sets.

2 Likes