Double free or corruption C++ Avro

Open
#53 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
cpp
Domain
backend

Research direction

Start with the supplied main function and reproduce the failure around avro::GenericDatum in json2avro. Read examples/kafka-serdes-avro-console-producer.cpp and src/avro/fh_price_t.json to compare the expected Avro setup with this example. Done means identifying the ownership or lifetime issue and confirming the reproducer no longer reports double-free or corruption.

Written by the indexing model from the issue text.

Description

I need to serialize a JSON string. I used the example code

./examples/kafka-serdes-avro-console-producer.cpp

to write a main, which serializes a JSON string as follows.

{"s":"BTCUSDT","c":"spread","u":123456789,"a":20000.4,"b":21000.5,"A":1.25,"B":0.58,"p":-1,"P":-1,"st":-1,"rt":1675000000123000}

Here is the code


#include "fh_price_t.h"
#include "avro/Encoder.hh"
#include "avro/Decoder.hh"
#include <avro/Generic.hh>
#include <avro/Specific.hh>
#include <avro/Exception.hh>

#include "libserdes/serdescpp-avro.h"

#include <iostream>
#include <sstream>
#include <string>
#include <fstream>

#define FATAL(reason...) do { \
    std::cerr << "% FATAL: " << reason << std::endl; \
    exit(1); \
} while (0)


template<class T> std::string to_string_a (const T& x) {
    std::ostringstream oss;
    oss << x;
    return oss.str();
}

/**
 * Convert JSON to Avro Datum.
 *
 * Returns 0 on success or -1 on failure.
 */
int json2avro (Serdes::Schema *schema, const std::string &json,
                      avro::GenericDatum **datump) {

    avro::ValidSchema *avro_schema = schema->object();

    /* Input stream from json string */
    std::istringstream iss(json);
    auto json_is = avro::istreamInputStream(iss);

    /* JSON decoder */
    avro::DecoderPtr json_decoder = avro::jsonDecoder(*avro_schema);
    avro::GenericDatum *datum = new avro::GenericDatum(*avro_schema);

    try {
        /* Decode JSON to Avro datum */
        json_decoder->init(*json_is);
        avro::decode(*json_decoder, *datum);

    } catch (const avro::Exception &e) {
        std::cerr << "% JSON to Avro transformation failed: "
                << e.what() << std::endl;
        return -1;
    }

    *datump = datum;

}

int main (int argc, char* argv) {

    izycoinstestavro::fh_price_t price{};
    price.s = "BTCUSDT";
    price.c = "spread";
    price.u = 123456789;
    price.a = 20000.4;
    price.A = 1.25;
    price.b = 21000.52;
    price.B = 0.58;
    price.p = -1;
    price.P = -1;
    price.st = -1;
    price.rt = 1675000000123000;

    std::ifstream file_ifstream("../src/avro/fh_price_t.json");
    std::stringstream buffer;
    buffer << file_ifstream.rdbuf();
    std::string schema_json = buffer.str();

    Serdes::Conf *sconf = Serdes::Conf::create();
    Serdes::Schema *schema;
    std::string errstr;

    if (sconf->set("schema.registry.url", "http://localhost:8081", errstr))
        FATAL("Conf failed: " << errstr);

    if (sconf->set("serializer.framing", "cp1", errstr))
        FATAL("Conf failed: " << errstr);

    Serdes::Avro *serdes = Serdes::Avro::create(sconf, errstr);
    if (!serdes)
        FATAL("Failed to create Serdes handle: " << errstr);

    schema = Serdes::Schema::add(serdes, "fh_price_t", schema_json, errstr);
    if (!schema)
        FATAL("Failed to register schema " << "fh_price_t" << ": " << errstr);

    std::cout << "% Registered schema " << schema->name() << " with id " << schema->id() << std::endl;


    avro::GenericDatum *datum = NULL;
    std::vector<char> out;

    /* Convert JSON to Avro object */
    std::string line = to_string_a(price);
    json2avro(schema, line, &datum);
    
    //std::cout << to_string_a(price) << std::endl;
    /*if (rr == -1) {
        FATAL("Failed to convert JSON to Avro " << to_string_a(price) << ": " << errstr);
    }

    // Serialize Avro 
    if (serdes->serialize(schema, datum, out, errstr) == -1) {
        std::cerr << "% Avro serialization failed: " << errstr << std::endl;
        delete datum;
        exit(1);
    }
    delete datum;

    std::cout << "Data Size : " << out.size() << std::endl;*/

    return 0;
}

When I run it, I get a double free or corruption (out) error. The error occurs at the output of the function. I have located the line that causes the error (because when I remove it, the error disappears)
avro::GenericDatum *datum = new avro::GenericDatum(*avro_schema);

I would like to know why and how to solve it.

Dominant language
C
Stars
12
Forks
62
PR merge metrics
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from confluentinc/libserdes

All issues in confluentinc/libserdes

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.