Shell命令数据迁移实践

中奖纪录侧

数据量：千万级

实施方案：数据库层面迁移，新数据一一匹配

实施脚本：

1、sql整理出需要导出的数据：

 

select a,b,c,d,e,f,g from table_1 into outfile '/data/aaa/haha.csv' fields terminated by ',' optionally enclosed by '\'' lines terminated by '\r\n';

2、使用shell分割记录结果，分批导入

 

 

split -a 3 -l 20000 haha.csv target/table2_ for i in `ls -1 target ` do mysql -s -N -p'xxxxxxx' DPEmidas -e "load data infile '/data/aaa/target/$i' IGNORE into table table2 fields terminated by ',' optionally enclosed by '\'' lines terminated by '\r\n'" mv /data/aaa/target/$i /data/aaa/complete/ sleep 1 done

 

3、

获取 所有项目的git地址： git/config文件如下： [core] repositoryformatversion = 0 filemode = false bare = false logallrefupdates = true symlinks = false ignorecase = true [remote "origin"] url = ssh://git@git.dianpingoa.com/emidas/emidas-decorate.git fetch = +refs/heads/*:refs/remotes/origin/* [branch "master"] remote = origin merge = refs/heads/master [branch "feature/search"] remote = origin merge = refs/heads/feature/search [branch "beta"] remote = origin merge = refs/heads/beta 需要获取每一个url 那一行后面的地址，正则需要匹配 空格符 url = xxxx 具体正则 (^\s*)url = (.*)，sed命令中需要转义 \ $ * + ( ) sed -n 's/\(^\s*\)url = \(.*\)/\2/p' */.git/config > test.txt

https://blog.csdn.net/z69183787/article/details/53693739