FlatBuffers 64 for C++ (#7935)

* First working hack of adding 64-bit. Don't judge :)

* Made vector_downward work on 64 bit types

* vector_downward uses size_t, added offset64 to reflection

* cleaned up adding offset64 in parser

* Add C++ testing skeleton for 64-bit

* working test for CreateVector64

* working >2 GiB buffers

* support for large strings

* simplified CreateString<> to just provide the offset type

* generalize CreateVector template

* update test_64.afb due to upstream format change

* Added Vector64 type, which is just an alias for vector ATM

* Switch to Offset64 for Vector64

* Update for reflection bfbs output change

* Starting to add support for vector64 type in C++

* made a generic CreateVector that can handle different offsets and vector types

* Support for 32-vector with 64-addressing

* Vector64 basic builder + tests working

* basic support for json vector64 support

* renamed fields in test_64bit.fbs to better reflect their use

* working C++ vector64 builder

* Apply --annotate-sparse-vector to 64-bit tests

* Enable Vector64 for --annotate-sparse-vectors

* Merged from upstream

* Add `near_string` field for testing 32-bit offsets alongside

* keep track of where the 32-bit and 64-bit regions are for flatbufferbuilder

* move template<> outside class body for GCC

* update run.sh to build and run tests

* basic assertion for adding 64-bit offset at the wrong time

* started to separate `FlatBufferBuilder` into two classes, 1 64-bit aware, the other not

* add test for nested flatbuffer vector64, fix bug in alignment of big vectors

* fixed CreateDirect method by iterating by Offset64 first

* internal refactoring of flatbufferbuilder

* block not supported languages in the parser from using 64-bit

* evolution tests for adding a vector64 field

* conformity tests for adding/removing offset64 attributes

* ensure test is for a big buffer

* add parser error tests for `offset64` and `vector64` attributes

* add missing static that GCC only complains about

* remove stdint-uintn.h header that gets automatically added

* move 64-bit CalculateOffset internal

* fixed return size of EndVector

* various fixes on windows

* add SizeT to vector_downward

* minimze range of size changes in vector and builder

* reworked how tracking if 64-offsets are added

* Add ReturnT to EndVector

* small cleanups

* remove need for second Array definition

* combine IndirectHelpers into one definition

* started support for vector of struct

* Support for 32/64-vectors of structs + Offset64

* small cleanups

* add verification for vector64

* add sized prefix for 64-bit buffers

* add fuzzer for 64-bit

* add example of adding many vectors using a wrapper table

* run the new -bfbs-gen-embed logic on the 64-bit tests

* remove run.sh and fix cmakelist issue

* fixed bazel rules

* fixed some PR comments

* add 64-bit tests to cmakelist

src/annotated_binary_text_gen.cpp

@@ -1,6 +1,7 @@
 #include "annotated_binary_text_gen.h"
 
 #include <algorithm>
+#include <cstdint>
 #include <fstream>
 #include <ostream>
 #include <sstream>
@@ -36,6 +37,7 @@ static std::string ToString(const BinarySectionType type) {
     case BinarySectionType::Struct: return "struct";
     case BinarySectionType::String: return "string";
     case BinarySectionType::Vector: return "vector";
+    case BinarySectionType::Vector64: return "vector64";
     case BinarySectionType::Unknown: return "unknown";
     case BinarySectionType::Union: return "union";
     case BinarySectionType::Padding: return "padding";
@@ -44,7 +46,9 @@ static std::string ToString(const BinarySectionType type) {
 }
 
 static bool IsOffset(const BinaryRegionType type) {
-  return type == BinaryRegionType::UOffset || type == BinaryRegionType::SOffset;
+  return type == BinaryRegionType::UOffset ||
+         type == BinaryRegionType::SOffset ||
+         type == BinaryRegionType::UOffset64;
 }
 
 template<typename T> std::string ToString(T value) {
@@ -119,6 +123,9 @@ static std::string ToValueString(const BinaryRegion &region,
     case BinaryRegionType::UType: return ToValueString<uint8_t>(region, binary);
 
     // Handle Offsets separately, incase they add additional details.
+    case BinaryRegionType::UOffset64:
+      s += ToValueString<uint64_t>(region, binary);
+      break;
     case BinaryRegionType::UOffset:
       s += ToValueString<uint32_t>(region, binary);
       break;
@@ -368,7 +375,8 @@ static void GenerateSection(std::ostream &os, const BinarySection &section,
   // As a space saving measure, skip generating every vector element, just put
   // the first and last elements in the output. Skip the whole thing if there
   // are only three or fewer elements, as it doesn't save space.
-  if (section.type == BinarySectionType::Vector &&
+  if ((section.type == BinarySectionType::Vector ||
+       section.type == BinarySectionType::Vector64) &&
       !output_config.include_vector_contents && section.regions.size() > 4) {
     // Generate the length region which should be first.
     GenerateRegion(os, section.regions[0], section, binary, output_config);