on the same point, batch row updates would be awesome too.

I have thought about it before, but haven't implement it due to can't run callbacks.

:+1:

+1 for Batch insert

Is there still no way to accomplish this? I tried building a string of multiple queries and passed it to Exec or Raw but I get errors when using Exec. Raw just seems not to do anything.

INSERT IGNORE INTO img_info (name, filename, path, height,width, mime) VALUES ('a', 'a.jpg.jpg','../../imagedb/thumbs/getty360/a.jpg', '73','110', 'image/jpeg'), ('b', 'b.png.png','../../imagedb/thumbs/getty360/b.png', '73','110', 'image/jpeg'), ('c', 'c.png.png','../../imagedb/thumbs/getty360/c.png', '73','110', 'image/jpeg'), ('d', 'd.png.png','../../imagedb/thumbs/getty360/d.png', '73','110', 'image/jpeg');
INSERT IGNORE INTO pending_uploads (filename, status, created_at) VALUES ('a', 0, '2015-04-23 12:22:37.596460147 -0400 EDT'), ('b', 0, '2015-04-23 12:22:37.596473783 -0400 EDT'), ('c', 0, '2015-04-23 12:22:37.596482969 -0400 EDT'), ('d', 0, '2015-04-23 12:22:37.596489454 -0400 EDT');
INSERT IGNORE INTO metadata (MasterNumber, Prod_Title, Prod_Code, Caption, Keywords,UploadDate) VALUES ('a', 'Getty 360','getty360', '','', '2015-04-23 12:22:37.596323687 -0400 EDT'), ('b', 'Getty 360','getty360', '','', '2015-04-23 12:22:37.596470372 -0400 EDT'), ('c', 'Getty 360','getty360', '','', '2015-04-23 12:22:37.596480429 -0400 EDT'), ('d', 'Getty 360','getty360', '','', '2015-04-23 12:22:37.596487448 -0400 EDT');
&{<nil> Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INSERT IGNORE INTO pending_uploads (filename, status, created_at) VALUES ('a', 0' at line 2 0 map[] <nil> 0xc20805e280 0xc20806aa50 0xc208090d80 0 <nil> <nil> false  map[] map[]}

/Users/acasanova/projects/go/src/com.drleonardo/imagelib/models/services/gettyservice_test.go:117Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INSERT IGNORE INTO pending_uploads (filename, status, created_at) VALUES ('a', 0' at line 2

the actual sql works fine in mysql

for now I'll just use individual statements
Any suggestions?

+1

the problem is in the mysql db driver.
i already send a pull request for this capability.
https://github.com/go-sql-driver/mysql/pull/339

+1, any way to do it now?

:+1:

Waiting for this feature eagerly aswell.

:+1:

+1 for Batch Inserts

https://github.com/jinzhu/gorm/issues/920

great news guys!
i have not been following this for a while but i just realized today that the multi statement (batch update exec) capability has been merged to the go-sql-driver/mysql library
https://github.com/go-sql-driver/mysql/pull/411
proudly contributed to this feature.
should be able to turn on this capability by adding &multiStatements=true to the DSN

One of the things I'd really like is to be able to INSERT many values at once in a single query (for speed):

INSERT INTO table (a, b, c) VALUES (?, ?, ?), (?, ?, ?), (?, ?, ?)

... etc.

I realize there are some issues here with the fact that (for example) the PrimaryKey/ID wouldn't be returned for multiple rows in this case, and this would require a subsequent SELECT to get that/those. However, that could also just be a disclaimer when using this function in the documentation.

Would be nice to be able to pass [] interface {} to .Create() and do this automatically.

What do we need to do to start moving on this? I'd like to use gorm for a project but this is a blocker for me.

It seems Like we'd need to trigger the callbacks once for each object in the slice. The tricky part would be error handling and reporting which could either fail immediately or continue processing on errors.

@TylerBrock It's something you can work around by constructing the queries manually, but that kind of sucks.

@benguild what would that look like? I'm just starting out with Golang and Gorm but maybe providing a workaround solution example would help a lot of people in this thread out. Would you mind showing us how something like that would be done?

Is it just a single transaction that executes multiple inserts via Gorm or would you use the pg driver directly?

@TylerBrock Something like this: https://github.com/jinzhu/gorm/issues/920 (except for batch insert: http://stackoverflow.com/questions/5526917/how-to-do-a-batch-insert-in-mysql )

Yup of course, thanks @pjebs, the SQL syntax I don't have any questions about.

What I was hoping was to see was something where gorm's helpers can be used to construct the statement in a more programmatic fashion but basically the Raw command and SQL builder part of the docs cover it, thanks!

Yeah dropping down to raw SQL was what I meant. Just be aware that as I
mentioned the last INSERT ID won't be filled in for the models after INSERT
unless you do a manual batch SELECT thereafter. Not doing that could create
some undesirable, unexpected, or suboptimal behavior... which I'm guessing
is why this hasn't been tackled yet.

However it's incredibly suboptimal to run 50 queries instead of 1, for
example, so running a SELECT afterward isn't as big a deal in some cases. I
think the logic I used to handle these cases was to only use the batch
INSERT/SELECT format if at least 3 rows needed to be created. Otherwise it
just did two solo INSERTs if a SELECT would need to take place afterward to

get the IDs... since the solo INSERTs return them automatically.

iPhoneから送信

Is []interface{} a type? Can you have a slice of interfaces?

Wouldn't you want to reflect on the passed interface{} and behave differently if it's slice?

@sandalwing Yes it is a type... that's what I said here: https://github.com/jinzhu/gorm/issues/255#issuecomment-243997661

Would be nice to be able to pass [] interface {} to .Create() and do this automatically.

... So yeah, that would be a way to do it. However, a .BatchCreate() method would be an alternative considering the caveats with this as I've mentioned in previous comments.

Any updates on this?

@drgomesp How would you like to see it implemented? — Here's what I propose:

database.BatchCreate(something interface {}, populatePrimaryKeyValues bool) — Method to insert multiple rows. Will return an error if row(s) are missing struct dependencies that need to be INSERT-ed prior. Optionally SELECT primary keys after.

I like that idea, @benguild. I'll probably give it a try on implementing it (kind of need it urgently).

Just as an update, here's what I've found so far:

• To be able to perform the batch insert is not a particularly difficult thing. However, making sure the ID's are properly set after is an issue. In the case of MySQL, the InnoDB engine will return the first inserted value as the LAST_INSERT_ID, and we could safely rely on mysql_affected_rows to increment the values and populate all of them correctly – only because this engine makes sure the auto increment feature is locked and safely incremented for a batch insert. For the other engines, I'm not entirely sure it would work the same way.

• The Scope represents a unit of work in gorm, which for this feature would mean to be able to call the Scope::Fields() function and get the list of columns, but also to be able to have a way to get the multiple values passed to the scope – maybe a Values() function that is decoupled from the way to get the fields. Right now, the values are populated in the Fields() function and that's not good – would need to be refactored.

• The callbacks could also be a problem. How do we want to handle the relationship callbacks in the case of batch operations? I'm not sure about that – so a first implementation would make the relationship persist operations to fallback to the original strategy, but maybe that's not exactly what we want.

I'm still trying to implement this solution in a generic way, and I'll get back to this thread once I have an update on that.

Here's the update:

I've managed to implement batch insert (without all normal callbacks), with proper IDs being populated after the operation has succeeded. You can find the implementation in my fork, at the batch-callback branch.

func main() {
    type Foo struct {
        ID   int64  `gorm:"column:id;primary_key"`
        Text string `gorm:"column:text"`
    }

    db := connect()
    defer db.Close()

    foos := []*Foo{}

    for i := 0; i < 10000; i++ {
        foo := &Foo{Text: "Foo"}
        foos = append(foos, foo)
    }

    start := time.Now()
    for _, foo := range foos {
        db.Create(foo)
    }
    elapsed := time.Since(start)

    log.Printf("Multiple queries took %s", elapsed)

    foos = []*Foo{}

        for i := 10001; i <= 20000; i++ {
        foo := &Foo{Text: "Foo"}
        foos = append(foos, foo)
    }

    start = time.Now()
    db.CreateBatch(foos)
    elapsed = time.Since(start)

    log.Printf("Single query took %s", elapsed)
}

2017/05/11 08:13:27 Multiple queries took 10.361453551s
2017/05/11 08:13:27 Single query took 81.625904ms

For now, the batch insert works only for normal fields (non-relationship). There's an ongoing investigation from my side on what would be the best way to implement that.

I still need to do some work in order to get this PR merged here.

Are you sure that works as intended? I'd want to see some serious testing with multiple concurrent requests, etc.

@benguild we're working with a highly concurrent program at my workplace (which is why we had the need for the batch operation in the first place), and it has been working so far (it's being tested for about two weeks now).

I'm not sure if I trust any SQL server in this way, personally. If they changed their rules between versions this would instantly break?

I'm not sure what you mean, @benguild.

Didn't you say you're relying on MySQL to insert the rows in order, since it only returns the initial ID of the first row? What about ON DUPLICATE KEY UPDATE ... skipping rows to avoid the whole batch being a fail? You won't know which ones were INSERT'ed.

Everything runs in a transaction anyways. If one fails, others will too.

Everything runs in a transaction anyways. If one fails, others will too.

What? I don't think this is a conclusive statement.

Right... I didn't catch your first point. It's something to be improved, definitely. Would you want to jump in and help out?

Our solution is much more complex, in that it will do a standard INSERT for less than 3 rows, and otherwise a batch INSERT then run a subsequent SELECT to get the IDs ... if requested by the caller. (so there's a separate bool parameter that gets passed in for whether or not to update the included structs)

@benguild What's the current progress on this feature? Would love to be able to do bulk inserts ❤️

@lalarsson We're not using a GORM-based solution for this so we've coded our own thing. Check out the discussion above.

My concerns would be in differing behaviors between various transaction states and SQL-variants/versions. Unless the DB server is explicitly returning the newly inserted ID values, to me it seems like a mistake to hard-code assumption rules based on expected behaviors when people are most likely automatically updating their infrastructures and not paying attention to version differences.

I have write a extension on database/sql to support mysql batch insert. You can try it: sqlext. Use it like this:

type GroupMember struct {
    ID        int64
    GroupID   int64
    Type      int8
    Extra     []byte
    JoinTime  time.Time
}

var mydb *gorm.DB
var members = []GroupMember{}
sqlext.BatchInsert(mydb.DB(),members)

//generated SQL: INSERT INTO group_member (id, group_id, type, extra, join_time) VALUES (?,?,?,?,?), (?,?,?,?,?) ...

This feature would be useful as its quite a common use case in DB.

188

I don't like this idea because it won't run callbacks, and save associations...
I can imagine many people will misuse this feature when have it. So maybe better write your own sql to do this job?

@wenweih That's what I've done, and I agree the inconsistency with this behavior is probably a reason to stay away from it.

With that said, it seems like a lot of people need it for specific times, and it'd be better if we can write it once perhaps with a disclaimer.

+1

So still no update on this? It's really sad, that in order to add N values of the same type i have to iterate over an array and save those one after another...also i don't understand the issues related to implementing this feature. Just do not return any IDs, references and so on if that's somehow not possible. The Option to drop thousands of values in a single statement is enough. Most of the time it's not even the use case to have the IDs assigned back to the structs after such an insert...such a shame that there's a debate over it for more than 3 years already but still no action nor a statement from the dev. I'm really thankful for all the work you've put into this project cause it's really made my life easier and since it's open source i'm not really in a position to criticize you for not doing something for free, but this seems like a must-have feature and if you intend to do any further work on this project - this should be priority nr. 1

@JRice I think you'd benefit from reading this: https://gomakethings.com/open-source-etiquette/ If you'd like to have the feature in gorm or in a wrapper, feel free to contribute.

If you have performance sensitive code, then write the SQL manually. An ORM is built for convenience, not performance.

@JRice There are other ways for you to help besides with code. For example, you could research how other ORMs have best-solved this issue and present your findings as a constructive way of contributing to the discussion.

+1 to add this feature

For those who are still struggling with this...

Here are a few clear examples I found of how one might achieve this without a feature in GORM:

https://stackoverflow.com/questions/12486436/golang-how-do-i-batch-sql-statements-with-package-database-sql
https://stackoverflow.com/questions/25289593/bulk-upserts-within-a-sql-transaction-in-golang

...Also, the xorm package is similar to gorm (with caveats) and supports bulk inserts, converting from one to the other should be reasonably simple, if gorm isn't a dependency. Someone more motivated than I might dig into their code and mimic what they've done: https://github.com/go-xorm/xorm

+1

+1

+1 for bulk insert

+1

+1

+1 for this feature.

+1

+1

+1

Hey everyone, it’s great that you are interested in this feature. Please register that interest as an emoji reaction to the original post or other comments that you find add value.

Otherwise everyone on the list gets an email when people comment +1. Commenting that way is now considered deprecated and bad github etiquette.

+1

Maybe if everyone hitting +1 would write some code, this would be done by now 😂

+1

+1

+1

+1

+1

+1

+1

+1

+1

+1

+1

+1

+1

+1

+1

+1 please save me from having to run Create() 1000 times 😖

I ended up doing what this guy at stackoverflow explains.
Adapted to my Gorm code, it goes something like this:

sqlStr := "INSERT INTO table_name(col1, col2, col3) VALUES "
vals := []interface{}{}
const rowSQL = "(?, ?, ?)"
var inserts []string

for _, elem := range sliceElems {
    inserts = append(inserts, rowSQL)
    vals = append(vals, elem.Prop1, elem.Prop2, elem.Prop3)
}
sqlStr = sqlStr + strings.Join(inserts, ",")

if err := DB.Exec(sqlStr, vals...).Error; err != nil {
    // :-(
}

It's not pretty and it won't work for everybody, but at least I don't have to run an insert for each element.

+1

+1

If you want to use array of gorm model instance as input, use this function in general.

func BatchInsert(db *gorm.DB, objArr []interface{}) error {
    // If there is no data, nothing to do.
    if len(objArr) == 0 {
        return nil 
    }   

    mainObj := objArr[0]
    mainScope := db.NewScope(mainObj)
    mainFields := mainScope.Fields()
    quoted := make([]string, 0, len(mainFields))
    for i := range mainFields { 
        // If primary key has blank value (0 for int, "" for string, nil for interface ...), skip it.
        // If field is ignore field, skip it.
        if (mainFields[i].IsPrimaryKey && mainFields[i].IsBlank) || (mainFields[i].IsIgnored) {
            continue
        }   
        quoted = append(quoted, mainScope.Quote(mainFields[i].DBName))
    }   

    placeholdersArr := make([]string, 0, len(objArr))

    for _, obj := range objArr {
        scope := db.NewScope(obj)
        fields := scope.Fields()
        placeholders := make([]string, 0, len(fields))
        for i := range fields {
            if (fields[i].IsPrimaryKey && fields[i].IsBlank) || (fields[i].IsIgnored) {
                continue
            }   
            placeholders = append(placeholders, scope.AddToVars(fields[i].Field.Interface()))
        }   
        placeholdersStr := "(" + strings.Join(placeholders, ", ") + ")"
        placeholdersArr = append(placeholdersArr, placeholdersStr)
        // add real variables for the replacement of placeholders' '?' letter later.
        mainScope.SQLVars = append(mainScope.SQLVars, scope.SQLVars...)
    }   

    mainScope.Raw(fmt.Sprintf("INSERT INTO %s (%s) VALUES %s",
        mainScope.QuotedTableName(),
        strings.Join(quoted, ", "),
        strings.Join(placeholdersArr, ", "),
    ))  

    if _, err := mainScope.SQLDB().Exec(mainScope.SQL, mainScope.SQLVars...); err != nil {
        return err
    }   
    return nil
}

One thing you have to do is to change []GormModel to []interface{}.

+1 for this feature

+1

+1

+1

+1

卧槽，提了4年了还没实现……

+1

+1

+1

+1

卧槽，原来搞了半天是没实现啊

https://github.com/jinzhu/gorm/issues/255#issuecomment-414527283
这个方法是可用的，但是不会返回id。

If you want to use array of gorm model instance as input, use this function in general.

func BatchInsert(db *gorm.DB, objArr []interface{}) error {
    // If there is no data, nothing to do.
    if len(objArr) == 0 {
        return nil 
    }   

    mainObj := objArr[0]
    mainScope := db.NewScope(mainObj)
    mainFields := mainScope.Fields()
    quoted := make([]string, 0, len(mainFields))
    for i := range mainFields { 
        // If primary key has blank value (0 for int, "" for string, nil for interface ...), skip it.
        // If field is ignore field, skip it.
        if (mainFields[i].IsPrimaryKey && mainFields[i].IsBlank) || (mainFields[i].IsIgnored) {
            continue
        }   
        quoted = append(quoted, mainScope.Quote(mainFields[i].DBName))
    }   

    placeholdersArr := make([]string, 0, len(objArr))

    for _, obj := range objArr {
        scope := db.NewScope(obj)
        fields := scope.Fields()
        placeholders := make([]string, 0, len(fields))
        for i := range fields {
            if (fields[i].IsPrimaryKey && fields[i].IsBlank) || (fields[i].IsIgnored) {
                continue
            }   
            placeholders = append(placeholders, scope.AddToVars(fields[i].Field.Interface()))
        }   
        placeholdersStr := "(" + strings.Join(placeholders, ", ") + ")"
        placeholdersArr = append(placeholdersArr, placeholdersStr)
        // add real variables for the replacement of placeholders' '?' letter later.
        mainScope.SQLVars = append(mainScope.SQLVars, scope.SQLVars...)
    }   

    mainScope.Raw(fmt.Sprintf("INSERT INTO %s (%s) VALUES %s",
        mainScope.QuotedTableName(),
        strings.Join(quoted, ", "),
        strings.Join(placeholdersArr, ", "),
    ))  

    if _, err := mainScope.SQLDB().Exec(mainScope.SQL, mainScope.SQLVars...); err != nil {
        return err
    }   
    return nil
}

One thing you have to do is to change []GormModel to []interface{}.

Thank you. For PostgreSQL small change is needed:

            // was: placeholders = append(placeholders, scope.AddToVars(fields[i].Field.Interface()))
            placeholders = append(placeholders, mainScope.AddToVars(fields[i].Field.Interface()))

Reason: PostgreSQL placeholders have $1, $2 ... $n format. Error is occured without this change because of placeholder counter resets for any array element. In MySQL placeholders are "?" and this problem is absent.

#255 (comment)
这个方法是可用的，但是不会返回id。
@krpn @hxmhlt @yhmin84
I used this with mysql db,but get empty data insert, is there a problem?
the value of placeholdersArr is not expected

Mymistake! It worked!

+1

If you want to use array of gorm model instance as input, use this function in general.

func BatchInsert(db *gorm.DB, objArr []interface{}) error {
    // If there is no data, nothing to do.
    if len(objArr) == 0 {
        return nil 
    }   

    mainObj := objArr[0]
    mainScope := db.NewScope(mainObj)
    mainFields := mainScope.Fields()
    quoted := make([]string, 0, len(mainFields))
    for i := range mainFields { 
        // If primary key has blank value (0 for int, "" for string, nil for interface ...), skip it.
        // If field is ignore field, skip it.
        if (mainFields[i].IsPrimaryKey && mainFields[i].IsBlank) || (mainFields[i].IsIgnored) {
            continue
        }   
        quoted = append(quoted, mainScope.Quote(mainFields[i].DBName))
    }   

    placeholdersArr := make([]string, 0, len(objArr))

    for _, obj := range objArr {
        scope := db.NewScope(obj)
        fields := scope.Fields()
        placeholders := make([]string, 0, len(fields))
        for i := range fields {
            if (fields[i].IsPrimaryKey && fields[i].IsBlank) || (fields[i].IsIgnored) {
                continue
            }   
            placeholders = append(placeholders, scope.AddToVars(fields[i].Field.Interface()))
        }   
        placeholdersStr := "(" + strings.Join(placeholders, ", ") + ")"
        placeholdersArr = append(placeholdersArr, placeholdersStr)
        // add real variables for the replacement of placeholders' '?' letter later.
        mainScope.SQLVars = append(mainScope.SQLVars, scope.SQLVars...)
    }   

    mainScope.Raw(fmt.Sprintf("INSERT INTO %s (%s) VALUES %s",
        mainScope.QuotedTableName(),
        strings.Join(quoted, ", "),
        strings.Join(placeholdersArr, ", "),
    ))  

    if _, err := mainScope.SQLDB().Exec(mainScope.SQL, mainScope.SQLVars...); err != nil {
        return err
    }   
    return nil
}

One thing you have to do is to change []GormModel to []interface{}.

If you want to skip over struct and slice-of-structs fields on your main model, for example if your DB driver doesn't know how to serialize them for an update, you can modify the lines

if (mainFields[i].IsPrimaryKey && mainFields[i].IsBlank) || (mainFields[i].IsIgnored) {

and

if (fields[i].IsPrimaryKey && fields[i].IsBlank) || (fields[i].IsIgnored) {

to

if (mainField.IsPrimaryKey && mainField.IsBlank) || (mainField.IsIgnored) ||
    (mainField.Field.Kind() == reflect.Struct) ||
    (mainField.Field.Kind() == reflect.Slice && mainField.Field.Type().Elem().Kind() == reflect.Struct) {

and

if (field.IsPrimaryKey && field.IsBlank) || (field.IsIgnored) ||
    (field.Field.Kind() == reflect.Struct) ||
    (field.Field.Kind() == reflect.Slice && field.Field.Type().Elem().Kind() == reflect.Struct) {

respectively.

It might be worth pulling those conditionals out into a single helper function and calling it in both places.

+1

+1

https://github.com/jinzhu/gorm/issues/255#issuecomment-414527283

if the value object has CreatedAt and UpdatedAt, you can add lines which that
```golang
if fields[i].Name == "CreatedAt" && fields[i].IsBlank {
fields[i].Set(now)
}

if fields[i].Name == "UpdatedAt" && fields[i].IsBlank {
fields[i].Set(now)
}

+1

+1

+1

+1

+1

I've created a library for bulk insert.
https://github.com/t-tiger/gorm-bulk-insert

Simply by passing a struct of slice, creation is executed all at once. Checking constraints on bulk insert, set CreatedAt and UpdatedAt automatically, also you can specify columns to exclude.

Currently only supports MySQL, but it's possible to support other RDB as well.

+1

+1

More than 4 yrs without native solution =/

@Djumpen Someone should write up a PR for this feature after this long, and have it reviewed by the core contributors; I don't have enough need or time for this, so it won't be me, but we can't expect @jinzhu to do more free labor when no one else has stepped up from the community despite this being a commonly requested feature.

+1

+1

+1

Hey, Can I take up this issue? though it may take sometime (2-3 weeks) for me but I would surely love to help out.

+1

+1

+1

+1

255 (comment)

This method does not log, I have improved it, as follows

type OwnDb struct {
    *gorm.DB
}

func (db *OwnDb)BatchInsert(objArr []interface{}) (int64, error) {
    // If there is no data, nothing to do.
    if len(objArr) == 0 {
        return 0, errors.New("insert a slice length of 0")
    }

    mainObj := objArr[0]
    mainScope := db.NewScope(mainObj)
    mainFields := mainScope.Fields()
    quoted := make([]string, 0, len(mainFields))
    for i := range mainFields {
        // If primary key has blank value (0 for int, "" for string, nil for interface ...), skip it.
        // If field is ignore field, skip it.
        if (mainFields[i].IsPrimaryKey && mainFields[i].IsBlank) || (mainFields[i].IsIgnored) {
            continue
        }
        quoted = append(quoted, mainScope.Quote(mainFields[i].DBName))
    }

    placeholdersArr := make([]string, 0, len(objArr))

    for _, obj := range objArr {
        scope := db.NewScope(obj)
        fields := scope.Fields()
        placeholders := make([]string, 0, len(fields))
        for i := range fields {
            if (fields[i].IsPrimaryKey && fields[i].IsBlank) || (fields[i].IsIgnored) {
                continue
            }
            var vars  interface{}
            if (fields[i].Name == "CreatedAt" || fields[i].Name == "UpdatedAt") && fields[i].IsBlank {
                vars = gorm.NowFunc()
            } else {
                vars = fields[i].Field.Interface()
            }
            placeholders = append(placeholders, scope.AddToVars(vars))
        }
        placeholdersStr := "(" + strings.Join(placeholders, ", ") + ")"
        placeholdersArr = append(placeholdersArr, placeholdersStr)
        // add real variables for the replacement of placeholders' '?' letter later.
        mainScope.SQLVars = append(mainScope.SQLVars, scope.SQLVars...)
    }
    mainScope.Raw(fmt.Sprintf("INSERT INTO %s (%s) VALUES %s",
        mainScope.QuotedTableName(),
        strings.Join(quoted, ", "),
        strings.Join(placeholdersArr, ", "),
    ))
    //Execute and Log
    if err :=mainScope.Exec().DB().Error;err != nil {
        return 0,err
    }
    return mainScope.DB().RowsAffected,nil
}

Get an instance

        db,err := gorm.Open(c.Driver, uri)

    if err != nil {
        panic("Database connection failed")
    }

    db.LogMode(c.LogMode)

    DB := &OwnDb{DB:db}

Please support this!

@RoseEnd Thanks for this, can you open a PR with matching tests? Thanks.

I have changed
placeholders = append(placeholders, scope.AddToVars(fields[i].Field.Interface()))
to
placeholders = append(placeholders, mainScope.AddToVars(fields[i].Field.Interface()))

and now it works for me.

batch update from slice would be great as well

Is this issue being actively worked on?

+1 for this feature. Batch insert would be nice.

Do we have a way to do Upserts in batch mode?
in MySQL we can achieve the same by using
insert into table_name(col1, col2, col3) values (val1,val2,val3) on ON DUPLICATE KEY UPDATE (col1 = values(col1), col2 = values(col2), col3 = values(col3);

@jdaggula

// BulkInsertOnDuplicateUpdate db batch insert
func BulkInsertOnDuplicateUpdate(db *gorm.DB, objArr []interface{}, updates string) error {
    // If there is no data, nothing to do.
    if len(objArr) == 0 {
        return nil
    }

    mainObj := objArr[0]
    mainScope := db.NewScope(mainObj)
    mainFields := mainScope.Fields()
    quoted := make([]string, 0, len(mainFields))
    for i := range mainFields {
        // If primary key has blank value (0 for int, "" for string, nil for interface ...), skip it.
        // If field is ignore field, skip it.
        if (mainFields[i].IsPrimaryKey && mainFields[i].IsBlank) || (mainFields[i].IsIgnored) {
            continue
        }
        quoted = append(quoted, mainScope.Quote(mainFields[i].DBName))
    }

    placeholdersArr := make([]string, 0, len(objArr))

    for _, obj := range objArr {
        scope := db.NewScope(obj)
        fields := scope.Fields()
        placeholders := make([]string, 0, len(fields))
        for i := range fields {
            if (fields[i].IsPrimaryKey && fields[i].IsBlank) || (fields[i].IsIgnored) {
                continue
            }
            placeholders = append(placeholders, scope.AddToVars(fields[i].Field.Interface()))
        }
        placeholdersStr := "(" + strings.Join(placeholders, ", ") + ")"
        placeholdersArr = append(placeholdersArr, placeholdersStr)
        // add real variables for the replacement of placeholders' '?' letter later.
        mainScope.SQLVars = append(mainScope.SQLVars, scope.SQLVars...)
    }

    mainScope.Raw(fmt.Sprintf("INSERT INTO %s (%s) VALUES %s ON DUPLICATE KEY UPDATE %s",
        mainScope.QuotedTableName(),
        strings.Join(quoted, ", "),
        strings.Join(placeholdersArr, ", "),
        updates,
    ))

    if _, err := mainScope.SQLDB().Exec(mainScope.SQL, mainScope.SQLVars...); err != nil {
        return err
    }
    return nil
}

if err := bulkdb.BulkInsertOnDuplicateUpdate(db, chunk, "pid = values(pid), process = values(process), updated_at = values(updated_at), direct = values(direct), `long` = values(`long`), short = values(short), count = count+1"); err != nil {
            logs.Error(err.Error())
            return err
        }

Doesn't look like the insert limitations would be handled and could run into error depending on which type of db you're using. Maybe using the mainScope.DB().RowsAffected variable and splitting the bulk insert into multiple bulk depending on the max allowed row in the db would do but i'm not sure there's a variable like max_allow_packet for every kind of db or hardcode a lower limit of 10000 per batch maybe. Postgres is 65535 parameters max i'm not 100% sure about others.

@t-tiger it's great

pass in []interface{} to batch insert using db.Create([]interface{})

I am passing []interface{} but it's not able to fetch column values from interface{}

Because of the potential issue mentioned by @thejdavid above regarding the maximum number of insert parameters, I added splitting of the data into chunks (if necessary), so that the inserted parameters limit is never exceeded.

I also changed placeholders = append(placeholders, scope.AddToVars(vars)) to placeholders = append(placeholders, mainScope.AddToVars(vars)) and removed mainScope.SQLVars = append(mainScope.SQLVars, scope.SQLVars...) because the original solution didn't work for me (the same placeholders were being inserted ($1, $2 etc.), leading to an error when the query was executed.

The whole code is listed below.

type OwnDb struct {
    *gorm.DB
}

const maxNumberOfBatchParameters = 65535

func (db *OwnDb) BatchInsert(objArr []interface{}) (int64, error) {
    // If there is no data, nothing to do.
    if len(objArr) == 0 {
        return 0, errors.New("slice must not be empty")
    }

    numberOfFields := db.calculateNumberOfFields((objArr)[0])
    return db.batchInsertChunks(splitIntoChunks(objArr, numberOfFields))
}

func splitIntoChunks(objArr []interface{}, numberOfFields int) [][]interface{} {
    var chunks [][]interface{}

    chunkSize := int(math.Floor(float64(maxNumberOfBatchParameters / float32(numberOfFields))))
    numberOfObjects := len(objArr)

    if numberOfObjects < chunkSize {
        return [][]interface{}{objArr}
    }

    for i := 0; i < numberOfObjects; i += chunkSize {
        end := i + chunkSize

        if end > numberOfObjects {
            end = numberOfObjects
        }

        chunks = append(chunks, objArr[i:end])
    }

    return chunks
}

func (db *OwnDb) calculateNumberOfFields(obj interface{}) int {
    return len(db.NewScope(obj).Fields())
}

func (db *OwnDb) batchInsertChunks(chunks [][]interface{}) (int64, error) {
    var rowsAffected int64 = 0
    for _, chunk := range chunks {
        chunkRowsAffected, err := db.batchInsert(chunk)
        if err != nil {
            return 0, err
        }

        rowsAffected += chunkRowsAffected
    }

    return rowsAffected, nil
}

func (db *OwnDb) batchInsert(objArr []interface{}) (int64, error) {
    // If there is no data, nothing to do.
    if len(objArr) == 0 {
        return 0, errors.New("slice must not be empty")
    }

    mainObj := objArr[0]
    mainScope := db.NewScope(mainObj)
    mainFields := mainScope.Fields()
    quoted := make([]string, 0, len(mainFields))
    for i := range mainFields {
        // If primary key has blank value (0 for int, "" for string, nil for interface ...), skip it.
        // If field is ignore field, skip it.
        if (mainFields[i].IsPrimaryKey && mainFields[i].IsBlank) || (mainFields[i].IsIgnored) {
            continue
        }
        quoted = append(quoted, mainScope.Quote(mainFields[i].DBName))
    }

    placeholdersArr := make([]string, 0, len(objArr))

    for _, obj := range objArr {
        scope := db.NewScope(obj)
        fields := scope.Fields()

        placeholders := make([]string, 0, len(fields))
        for i := range fields {
            if (fields[i].IsPrimaryKey && fields[i].IsBlank) || (fields[i].IsIgnored) {
                continue
            }
            var vars interface{}
            if (fields[i].Name == "CreatedAt" || fields[i].Name == "UpdatedAt") && fields[i].IsBlank {
                vars = gorm.NowFunc()
            } else {
                vars = fields[i].Field.Interface()
            }
            placeholders = append(placeholders, mainScope.AddToVars(vars))
        }

        placeholdersStr := "(" + strings.Join(placeholders, ", ") + ")"
        placeholdersArr = append(placeholdersArr, placeholdersStr)
    }

    mainScope.Raw(fmt.Sprintf("INSERT INTO %s (%s) VALUES %s;",
        mainScope.QuotedTableName(),
        strings.Join(quoted, ", "),
        strings.Join(placeholdersArr, ", "),
    ))

    // Execute and Log
    if err := mainScope.Exec().DB().Error; err != nil {
        return 0, err
    }
    return mainScope.DB().RowsAffected, nil
}

Still no proper support for this ?

I tried to give a []interface{} as parameter to db.Save or db.Create. It panics with reflect: call of reflect.Value.Interface on zero Value

Still no proper support for this ?

I tried to give a []interface{} as parameter to db.Save or db.Create. It panics with reflect: call of reflect.Value.Interface on zero Value

me too.

+1
I think it is good to add this feature as well.

+1

+1

• bulk insert, bulk query with gorm
• Bulk insert optimisations using raw sql in Golang (mysql/psql)

+1

+1

+1

If anyone ends up here, I had a great experience by integrating https://github.com/t-tiger/gorm-bulk-insert with gorm.

Please implement bulk functions for common queries, including insert, update, and delete. In medium to large applications, batching becomes a first tier priority for maintaining efficient, scalable systems.

Just to remind you guys: don't forget to supportignore modifiers.

+1

+1

+1

@jinzhu Do you appreciate the +1 or do you keep track of the votes in the original post?

I would really like this and I can help too so I’m watching this issue but I’m kind of against all the +1. If it’s not the desired way according to the maintainers could we try to do a 👍 on the original post instead?

EDIT: I made gorm-bulk similar to the comments above and the one mentioned by t-tiger but with a (in my opinion) more flexible approach letting the end user define their own scope (SQL, and values). It also help you generate code to convert to the required interface slice. It has three default bulk actions bundled as of now (instead of just INSERT INTO) BulkInsert, BulkInsertIgnore and BulkInsertOnDuplicateKeyUpdate but you can define your own. Oh, and it also support gorm:insert_option tag. Not yet stable (no v1 release) .

Hey,
I'd really like to see this feature in the original gorm repo as well. Is there any reason not to add the implementation from @sljeff (provided he agrees to creating a PR)?

Wow! This feature originally open in 2014, +1ed by soooo many and still not closed. Whats stopping this?

+1 for batch insert. When I tried to put []interface{} as parameter to db.Save and get back this panic "reflect: call of reflect.Value.Interface on zero Value"

Added its support in v2, so going to close this loooooong issue.

Hey, @jinzhu thanks for this. Is v2 ready for use?

@King-Success it's on a branch: https://github.com/jinzhu/gorm/tree/v2_dev

pull it down and give it a whirl!

Thanks, @eldilibra

@King-Success it's on a branch: https://github.com/jinzhu/gorm/tree/v2_dev

pull it down and give it a whirl!

Hello! which function does the bulk insert in this branch? thanks!

Hello! which function does the bulk insert in this branch? thanks!

It's the regular methods you use for single row inserts. Just pass a slice of your type and gorm will create a bulk statement.

type MyType struct {
    Value string
}

func main() {
    // ...
    toInsert := []MyType{
        {Value: "first row"},
        {Value: "second row"},
        {Value: "third row"},
    }

    db.Create(toInsert)
}

Will create a query similar to

INSERT INTO "my_types" ("value") VALUES ('first row'),('second row'),('third row')

Hello

I can successfully use this func to insert single value :
```
func (d TaskDB) Create(task model.Task) (model.Task, error) {
err := d.cl.Create(task).Error
if err != nil {
return nil, err
}
return task, nil
}

But when I use this, it's become error

func (d TaskDB) Creates(tasks []model.Task) ([]model.Task, error) {
err := d.cl.Create(tasks).Error
if err != nil {
return nil, err
}
return tasks, nil
}
```

Anyone can help? thanks

This is what I did:

given these structs

type Item struct {
    Name string
}

type InputItem struct {
    Name string
}

part that bulk inserts a slice InputItem into Item:

var input []InputItem
var items []Item

tx := db.Begin()

defer func() {
    if r := recover(); r != nil {
        tx.Rollback()
    }
}()

if txErr := tx.Error; txErr != nil {
    panic(txErr)
}

for _, i := range input {
    item := Item{Name: i.Name}
    items = append(items, item)

    if createErr := tx.Create(&item).Error; createErr != nil {
        panic(createErr)
    }
}

if commitErr := tx.Commit().Error; commitErr != nil { 
    panic(commitErr)
}

how to implement bulk insert with ignore??? like insert ignore into...

@easy-money-sniper For v1, have a look at gorm-bulk and the feature BulkInsertIgnore().

Is it possible to scan back the ids of the bulk inserted records (like in case of a single record)? I tried Scan(&[]recordType{}) and ScanRows(), but they doesn't seem to work ("throws reflection error, "comparing with nil").

Is it possible to scan back the ids of the bulk inserted records (like in case of a single record)? I tried Scan(&[]recordType{}) and ScanRows(), but they doesn't seem to work ("throws reflection error, "comparing with nil").

@dark-shade Yes, will scan back ids

how to implement bulk insert with ignore??? like insert ignore into...

@easy-money-sniper

db.Clauses(clause.OnConflict{DoNothing: true}).Create(&tasks)