Added accessor in Java to get vectors as ByteBuffers.

Also cleaned up ByteBuffer usage in general: ByteBuffer.position
now universally indicates the start of a ByteBuffer.

Change-Id: Ic4bfb98f9df9501b8fc82de2c45db7d7311135ac
Tested: on Linux.

docs/html/md__java_usage.html

@@ -60,10 +60,11 @@ Monster monster = Monster.getRootAsMonster(bb);
 Vec3 pos = monster.pos();
 </pre><p>Note that whenever you access a new object like in the <code>pos</code> example above, a new temporary accessor object gets created. If your code is very performance sensitive (you iterate through a lot of objects), there's a second <code>pos()</code> method to which you can pass a <code>Vec3</code> object you've already created. This allows you to reuse it across many calls and reduce the amount of object allocation (and thus garbage collection) your program does.</p>
 <p>Java does not support unsigned scalars. This means that any unsigned types you use in your schema will actually be represented as a signed value. This means all bits are still present, but may represent a negative value when used. For example, to read a <code>byte b</code> as an unsigned number, you can do: <code>(short)(b &amp; 0xFF)</code></p>
-<p>Sadly the string accessors currently always create a new string when accessed, since FlatBuffer's UTF-8 strings can't be read in-place by Java.</p>
+<p>The default string accessor (e.g. <code>monster.name()</code>) currently always create a new Java <code>String</code> when accessed, since FlatBuffer's UTF-8 strings can't be used in-place by <code>String</code>. Alternatively, use <code>monster.nameAsByteBuffer()</code> which returns a <code>ByteBuffer</code> referring to the UTF-8 data in the original <code>ByteBuffer</code>, which is much more efficient. The <code>ByteBuffer</code>'s <code>position</code> points to the first character, and its <code>limit</code> to just after the last.</p>
 <p>Vector access is also a bit different from C++: you pass an extra index to the vector field accessor. Then a second method with the same name suffixed by <code>Length</code> let's you know the number of elements you can access: </p><pre class="fragment">for (int i = 0; i &lt; monster.inventoryLength(); i++)
     monster.inventory(i); // do something here
-</pre><p>If you specified a file_indentifier in the schema, you can query if the buffer is of the desired type before accessing it using: </p><pre class="fragment">if (Monster.MonsterBufferHasIdentifier(bb, start)) ...
+</pre><p>Alternatively, much like strings, you can use <code>monster.inventoryAsByteBuffer()</code> to get a <code>ByteBuffer</code> referring to the whole vector. Use <code>ByteBuffer</code> methods like <code>asFloatBuffer</code> to get specific views if needed.</p>
+<p>If you specified a file_indentifier in the schema, you can query if the buffer is of the desired type before accessing it using: </p><pre class="fragment">if (Monster.MonsterBufferHasIdentifier(bb)) ...
 </pre><h2>Buffer construction in Java</h2>
 <p>You can also construct these buffers in Java using the static methods found in the generated code, and the FlatBufferBuilder class: </p><pre class="fragment">FlatBufferBuilder fbb = new FlatBufferBuilder();
 </pre><p>Create strings: </p><pre class="fragment">int str = fbb.createString("MyMonster");
@@ -86,7 +87,7 @@ int inv = fbb.endVector();
 </pre><p>You can use the generated method <code>startInventoryVector</code> to conveniently call <code>startVector</code> with the right element size. You pass the number of elements you want to write. You write the elements backwards since the buffer is being constructed back to front.</p>
 <p>There are <code>add</code> functions for all the scalar types. You use <code>addOffset</code> for any previously constructed objects (such as other tables, strings, vectors). For structs, you use the appropriate <code>create</code> function in-line, as shown above in the <code>Monster</code> example.</p>
 <p>To finish the buffer, call: </p><pre class="fragment">Monster.finishMonsterBuffer(fbb, mon);
-</pre><p>The buffer is now ready to be transmitted. It is contained in the <code>ByteBuffer</code> which you can obtain from <code>fbb.dataBuffer()</code>. Importantly, the valid data does not start from offset 0 in this buffer, but from <code>fbb.dataStart()</code> (this is because the data was built backwards in memory). It ends at <code>fbb,capacity()</code>.</p>
+</pre><p>The buffer is now ready to be transmitted. It is contained in the <code>ByteBuffer</code> which you can obtain from <code>fbb.dataBuffer()</code>. Importantly, the valid data does not start from offset 0 in this buffer, but from <code>fbb.dataBuffer().position()</code> (this is because the data was built backwards in memory). It ends at <code>fbb.capacity()</code>.</p>
 <h2>Text Parsing</h2>
 <p>There currently is no support for parsing text (Schema's and JSON) directly from Java, though you could use the C++ parser through JNI. Please see the C++ documentation for more on text parsing. </p>
 </div></div><!-- contents -->

docs/source/JavaUsage.md

@@ -1,7 +1,7 @@
 # Use in Java
 
-FlatBuffers supports reading and writing binary FlatBuffers in Java. Generate code
+FlatBuffers supports reading and writing binary FlatBuffers in Java. Generate
-for Java with the `-j` option to `flatc`.
+code for Java with the `-j` option to `flatc`.
 
 See `javaTest.java` for an example. Essentially, you read a FlatBuffer binary
 file into a `byte[]`, which you then turn into a `ByteBuffer`, which you pass to
@@ -19,8 +19,8 @@ Note that whenever you access a new object like in the `pos` example above,
 a new temporary accessor object gets created. If your code is very performance
 sensitive (you iterate through a lot of objects), there's a second `pos()`
 method to which you can pass a `Vec3` object you've already created. This allows
-you to reuse it across many calls and reduce the amount of object allocation (and
+you to reuse it across many calls and reduce the amount of object allocation
-thus garbage collection) your program does.
+(and thus garbage collection) your program does.
 
 Java does not support unsigned scalars. This means that any unsigned types you
 use in your schema will actually be represented as a signed value. This means
@@ -28,8 +28,12 @@ all bits are still present, but may represent a negative value when used.
 For example, to read a `byte b` as an unsigned number, you can do:
 `(short)(b & 0xFF)`
 
-Sadly the string accessors currently always create a new string when accessed,
+The default string accessor (e.g. `monster.name()`) currently always create
-since FlatBuffer's UTF-8 strings can't be read in-place by Java.
+a new Java `String` when accessed, since FlatBuffer's UTF-8 strings can't be
+used in-place by `String`. Alternatively, use `monster.nameAsByteBuffer()`
+which returns a `ByteBuffer` referring to the UTF-8 data in the original
+`ByteBuffer`, which is much more efficient. The `ByteBuffer`'s `position`
+points to the first character, and its `limit` to just after the last.
 
 Vector access is also a bit different from C++: you pass an extra index
 to the vector field accessor. Then a second method with the same name
@@ -38,10 +42,14 @@ suffixed by `Length` let's you know the number of elements you can access:
     for (int i = 0; i < monster.inventoryLength(); i++)
         monster.inventory(i); // do something here
 
+Alternatively, much like strings, you can use `monster.inventoryAsByteBuffer()`
+to get a `ByteBuffer` referring to the whole vector. Use `ByteBuffer` methods
+like `asFloatBuffer` to get specific views if needed.
+
 If you specified a file_indentifier in the schema, you can query if the
 buffer is of the desired type before accessing it using:
 
-    if (Monster.MonsterBufferHasIdentifier(bb, start)) ...
+    if (Monster.MonsterBufferHasIdentifier(bb)) ...
 
 
 ## Buffer construction in Java
@@ -105,8 +113,9 @@ To finish the buffer, call:
 
 The buffer is now ready to be transmitted. It is contained in the `ByteBuffer`
 which you can obtain from `fbb.dataBuffer()`. Importantly, the valid data does
-not start from offset 0 in this buffer, but from `fbb.dataStart()` (this is
+not start from offset 0 in this buffer, but from `fbb.dataBuffer().position()`
-because the data was built backwards in memory). It ends at `fbb,capacity()`.
+(this is because the data was built backwards in memory).
+It ends at `fbb.capacity()`.
 
 
 ## Text Parsing

java/flatbuffers/FlatBufferBuilder.java

@@ -245,6 +245,7 @@ public class FlatBufferBuilder {
     public void finish(int root_table) {
         prep(minalign, SIZEOF_INT);
         addOffset(root_table);
+        bb.position(space);
     }
 
     public void finish(int root_table, String file_identifier) {
@@ -255,13 +256,20 @@ public class FlatBufferBuilder {
         for (int i = FILE_IDENTIFIER_LENGTH - 1; i >= 0; i--) {
             addByte((byte)file_identifier.charAt(i));
         }
-        addOffset(root_table);
+        finish(root_table);
     }
 
+    // Get the ByteBuffer representing the FlatBuffer. Only call this after you've
+    // called finish(). The actual data starts at the ByteBuffer's current position,
+    // not necessarily at 0.
     public ByteBuffer dataBuffer() { return bb; }
 
-    // The FlatBuffer data doesn't start at offset 0 in the ByteBuffer:
+    // The FlatBuffer data doesn't start at offset 0 in the ByteBuffer,
-    public int dataStart() {
+    // but now the ByteBuffer's position is set to that location upon
+    // finish(). This method should not be needed anymore, but is left
+    // here as private for the moment to document this API change.
+    // It will be removed in the future.
+    private int dataStart() {
         return space;
     }
 
@@ -273,7 +281,7 @@ public class FlatBufferBuilder {
     }
 
     // Utility function for copying a byte array that starts at 0.
-    public byte[] sizedByteArray(){
+    public byte[] sizedByteArray() {
         return sizedByteArray(space, bb.capacity() - space);
     }
 }

java/flatbuffers/Table.java

@@ -38,6 +38,10 @@ public class Table {
   }
 
   // Create a java String from UTF-8 data stored inside the flatbuffer.
+  // This allocates a new string and converts to wide chars upon each access,
+  // which is not very efficient. Instead, each FlatBuffer string also comes with an
+  // accessor based on __vector_as_bytebuffer below, which is much more efficient,
+  // assuming your Java program can handle UTF-8 data directly.
   protected String __string(int offset) {
     offset += bb.getInt(offset);
     if (bb.hasArray()) {
@@ -45,10 +49,11 @@ public class Table {
     } else {
       // We can't access .array(), since the ByteBuffer is read-only.
       // We're forced to make an extra copy:
-      bb.position(offset + SIZEOF_INT);
       byte[] copy = new byte[bb.getInt(offset)];
+      int old_pos = bb.position();
+      bb.position(offset + SIZEOF_INT);
       bb.get(copy);
-      bb.position(0);
+      bb.position(old_pos);
       return new String(copy, 0, copy.length, Charset.forName("UTF-8"));
     }
   }
@@ -66,6 +71,21 @@ public class Table {
     return offset + bb.getInt(offset) + SIZEOF_INT;  // data starts after the length
   }
 
+  // Get a whole vector as a ByteBuffer. This is efficient, since it only allocates a new
+  // bytebuffer object, but does not actually copy the data, it still refers to the same
+  // bytes as the original ByteBuffer.
+  // Also useful with nested FlatBuffers etc.
+  protected ByteBuffer __vector_as_bytebuffer(int vector_offset, int elem_size) {
+    int o = __offset(vector_offset);
+    if (o == 0) return null;
+    int old_pos = bb.position();
+    bb.position(__vector(o));
+    ByteBuffer nbb = bb.slice();
+    bb.position(old_pos);
+    nbb.limit(__vector_len(o) * elem_size);
+    return nbb;
+  }
+
   // Initialize any Table-derived type to point to the union at the given offset.
   protected Table __union(Table t, int offset) {
     offset += bb_pos;
@@ -74,12 +94,12 @@ public class Table {
     return t;
   }
 
-  protected static boolean __has_identifier(ByteBuffer bb, int offset, String ident) {
+  protected static boolean __has_identifier(ByteBuffer bb, String ident) {
     if (ident.length() != FILE_IDENTIFIER_LENGTH)
         throw new AssertionError("FlatBuffers: file identifier must be length " +
                                  FILE_IDENTIFIER_LENGTH);
     for (int i = 0; i < FILE_IDENTIFIER_LENGTH; i++) {
-      if (ident.charAt(i) != (char)bb.get(offset + SIZEOF_INT + i)) return false;
+      if (ident.charAt(i) != (char)bb.get(bb.position() + SIZEOF_INT + i)) return false;
     }
     return true;
   }

src/idl_gen_java.cpp

@@ -174,16 +174,16 @@ static void GenStruct(const Parser &parser, StructDef &struct_def,
     // of a FlatBuffer
     code += "  public static " + struct_def.name + " getRootAs";
     code += struct_def.name;
-    code += "(ByteBuffer _bb, int offset) { ";
+    code += "(ByteBuffer _bb) { ";
     code += "_bb.order(ByteOrder.LITTLE_ENDIAN); ";
     code += "return (new " + struct_def.name;
-    code += "()).__init(_bb.getInt(offset) + offset, _bb); }\n";
+    code += "()).__init(_bb.getInt(_bb.position()) + _bb.position(), _bb); }\n";
     if (parser.root_struct_def == &struct_def) {
       if (parser.file_identifier_.length()) {
         // Check if a buffer has the identifier.
         code += "  public static boolean " + struct_def.name;
-        code += "BufferHasIdentifier(ByteBuffer _bb, int offset) { return ";
+        code += "BufferHasIdentifier(ByteBuffer _bb) { return ";
-        code += "__has_identifier(_bb, offset, \"" + parser.file_identifier_;
+        code += "__has_identifier(_bb, \"" + parser.file_identifier_;
         code += "\"); }\n";
       }
     }
@@ -285,6 +285,15 @@ static void GenStruct(const Parser &parser, StructDef &struct_def,
       code += offset_prefix;
       code += "__vector_len(o) : 0; }\n";
     }
+    if (field.value.type.base_type == BASE_TYPE_VECTOR ||
+        field.value.type.base_type == BASE_TYPE_STRING) {
+      code += "  public ByteBuffer " + MakeCamel(field.name, false);
+      code += "AsByteBuffer() { return __vector_as_bytebuffer(";
+      code += NumToString(field.value.offset) + ", ";
+      code += NumToString(field.value.type.base_type == BASE_TYPE_STRING ? 1 :
+                          InlineSize(field.value.type.VectorType()));
+      code += "); }\n";
+    }
   }
   code += "\n";
   if (struct_def.fixed) {

tests/JavaTest.java

@@ -41,7 +41,7 @@ class JavaTest {
         // Now test it:
 
         ByteBuffer bb = ByteBuffer.wrap(data);
-        TestBuffer(bb, 0);
+        TestBuffer(bb);
 
         // Second, let's create a FlatBuffer from scratch in Java, and test it also.
         // We use an initial size of 1 to exercise the reallocation algorithm,
@@ -95,7 +95,7 @@ class JavaTest {
         try {
              DataOutputStream os = new DataOutputStream(new FileOutputStream(
                                            "monsterdata_java_wire.bin"));
-             os.write(fbb.dataBuffer().array(), fbb.dataStart(), fbb.offset());
+             os.write(fbb.dataBuffer().array(), fbb.dataBuffer().position(), fbb.offset());
              os.close();
         } catch(java.io.IOException e) {
             System.out.println("FlatBuffers test: couldn't write file");
@@ -103,20 +103,20 @@ class JavaTest {
         }
 
         // Test it:
-        TestBuffer(fbb.dataBuffer(), fbb.dataStart());
+        TestBuffer(fbb.dataBuffer());
 
         // Make sure it also works with read only ByteBuffers. This is slower,
         // since creating strings incurs an additional copy
         // (see Table.__string).
-        TestBuffer(fbb.dataBuffer().asReadOnlyBuffer(), fbb.dataStart());
+        TestBuffer(fbb.dataBuffer().asReadOnlyBuffer());
 
         System.out.println("FlatBuffers test: completed successfully");
     }
 
-    static void TestBuffer(ByteBuffer bb, int start) {
+    static void TestBuffer(ByteBuffer bb) {
-        TestEq(Monster.MonsterBufferHasIdentifier(bb, start), true);
+        TestEq(Monster.MonsterBufferHasIdentifier(bb), true);
 
-        Monster monster = Monster.getRootAsMonster(bb, start);
+        Monster monster = Monster.getRootAsMonster(bb);
 
         TestEq(monster.hp(), (short)80);
         TestEq(monster.mana(), (short)150);  // default
@@ -145,6 +145,13 @@ class JavaTest {
             invsum += monster.inventory(i);
         TestEq(invsum, 10);
 
+        // Alternative way of accessing a vector:
+        ByteBuffer ibb = monster.inventoryAsByteBuffer();
+        invsum = 0;
+        while (ibb.position() < ibb.limit())
+            invsum += ibb.get();
+        TestEq(invsum, 10);
+
         Test test_0 = monster.test4(0);
         Test test_1 = monster.test4(1);
         TestEq(monster.test4Length(), 2);

tests/MyGame/Example/Monster.java

@@ -8,8 +8,8 @@ import java.util.*;
 import flatbuffers.*;
 
 public class Monster extends Table {
-  public static Monster getRootAsMonster(ByteBuffer _bb, int offset) { _bb.order(ByteOrder.LITTLE_ENDIAN); return (new Monster()).__init(_bb.getInt(offset) + offset, _bb); }
+  public static Monster getRootAsMonster(ByteBuffer _bb) { _bb.order(ByteOrder.LITTLE_ENDIAN); return (new Monster()).__init(_bb.getInt(_bb.position()) + _bb.position(), _bb); }
-  public static boolean MonsterBufferHasIdentifier(ByteBuffer _bb, int offset) { return __has_identifier(_bb, offset, "MONS"); }
+  public static boolean MonsterBufferHasIdentifier(ByteBuffer _bb) { return __has_identifier(_bb, "MONS"); }
   public Monster __init(int _i, ByteBuffer _bb) { bb_pos = _i; bb = _bb; return this; }
 
   public Vec3 pos() { return pos(new Vec3()); }
@@ -17,24 +17,30 @@ public class Monster extends Table {
   public short mana() { int o = __offset(6); return o != 0 ? bb.getShort(o + bb_pos) : 150; }
   public short hp() { int o = __offset(8); return o != 0 ? bb.getShort(o + bb_pos) : 100; }
   public String name() { int o = __offset(10); return o != 0 ? __string(o + bb_pos) : null; }
+  public ByteBuffer nameAsByteBuffer() { return __vector_as_bytebuffer(10, 1); }
   public byte inventory(int j) { int o = __offset(14); return o != 0 ? bb.get(__vector(o) + j * 1) : 0; }
   public int inventoryLength() { int o = __offset(14); return o != 0 ? __vector_len(o) : 0; }
+  public ByteBuffer inventoryAsByteBuffer() { return __vector_as_bytebuffer(14, 1); }
   public byte color() { int o = __offset(16); return o != 0 ? bb.get(o + bb_pos) : 8; }
   public byte testType() { int o = __offset(18); return o != 0 ? bb.get(o + bb_pos) : 0; }
   public Table test(Table obj) { int o = __offset(20); return o != 0 ? __union(obj, o) : null; }
   public Test test4(int j) { return test4(new Test(), j); }
   public Test test4(Test obj, int j) { int o = __offset(22); return o != 0 ? obj.__init(__vector(o) + j * 4, bb) : null; }
   public int test4Length() { int o = __offset(22); return o != 0 ? __vector_len(o) : 0; }
+  public ByteBuffer test4AsByteBuffer() { return __vector_as_bytebuffer(22, 4); }
   public String testarrayofstring(int j) { int o = __offset(24); return o != 0 ? __string(__vector(o) + j * 4) : null; }
   public int testarrayofstringLength() { int o = __offset(24); return o != 0 ? __vector_len(o) : 0; }
+  public ByteBuffer testarrayofstringAsByteBuffer() { return __vector_as_bytebuffer(24, 4); }
   /// an example documentation comment: this will end up in the generated code multiline too
   public Monster testarrayoftables(int j) { return testarrayoftables(new Monster(), j); }
   public Monster testarrayoftables(Monster obj, int j) { int o = __offset(26); return o != 0 ? obj.__init(__indirect(__vector(o) + j * 4), bb) : null; }
   public int testarrayoftablesLength() { int o = __offset(26); return o != 0 ? __vector_len(o) : 0; }
+  public ByteBuffer testarrayoftablesAsByteBuffer() { return __vector_as_bytebuffer(26, 4); }
   public Monster enemy() { return enemy(new Monster()); }
   public Monster enemy(Monster obj) { int o = __offset(28); return o != 0 ? obj.__init(__indirect(o + bb_pos), bb) : null; }
   public byte testnestedflatbuffer(int j) { int o = __offset(30); return o != 0 ? bb.get(__vector(o) + j * 1) : 0; }
   public int testnestedflatbufferLength() { int o = __offset(30); return o != 0 ? __vector_len(o) : 0; }
+  public ByteBuffer testnestedflatbufferAsByteBuffer() { return __vector_as_bytebuffer(30, 1); }
   public Monster testempty() { return testempty(new Monster()); }
   public Monster testempty(Monster obj) { int o = __offset(32); return o != 0 ? obj.__init(__indirect(o + bb_pos), bb) : null; }